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Executive  Summary 


The  discipline  of  software  process  management  has  grown  remark¬ 
ably  since  the  Software  Engineering  Institute  (SEI)  introduced  the 
concept  in  1985.  Since  then,  our  work  in  software  development 
process  has  spurred  the  growth  of  a  worldwide  market,  connecting 
researchers,  tool  developers,  consultants,  educators,  and  software  pro¬ 
fessionals.  Today,  de  facto  and  de  jure  standards  continue  to  emerge, 
signaling  a  maturing  of  the  field.  Yet,  new  opportunities  and  new 
challenges  are  already  before  us,  extending  the  discipline  of  process 
management  into  exciting  new  areas.  From  software-intensive  sys¬ 
tems  to  the  software-dependent  enterprise,  from  health  care  to  agricul¬ 
ture,  and  from  technology  acquirers  to  service  providers,  the  SEI  has 
seen  growing  demand  to  bring  the  benefits  of  process  management  to 
a  diversity  of  software-critical  domains. 

To  prepare  for  the  road  ahead,  the  SEI  undertook  a  collaborative  effort 
with  other  leaders  in  the  process  community  to  explore  process  needs 
for  today,  the  foreseeable  future,  and  the  unforeseeable  future.  In 
August  of  2004,  the  SEI  launched  the  International  Process  Research 
Consortium  (IPRC)  as  the  primary  forum  for  accomplishing  this  task. 
Through  a  series  of  six  workshops  of  intensive  discussion,  debate, 
imagination,  and  expertise,  the  28  members  of  the  IPRC,  along  with 
the  support  of  several  notable  and  objective  reviewers,  developed  this 
Process  Research  Framework.  It  is  a  thematic  guide  to  critical  process 
research.  It  is  wide-ranging  and  far-reaching.  And  while  the  primary 
audience  is  the  membership  of  the  IPRC  and  the  organizations  and 
constituents  they  represent,  it  is  our  sincere  hope  that  it  will  also 
prove  valuable  to  those  with  the  drive,  the  influence,  and  the  vision  to 
turn  research  into  solutions. 

This  framework  encompasses  our  thinking  on  emerging  technology 
and  four  research  themes: 

•  Process  and  Product  Quality  Relationships 

•  Process  Engineering 

•  Managing  Project  Processes 

•  Process  Deployment 

The  themes  comprise  20  research  nodes  that  address  more  than  230 
research  questions.  Underpinning  the  themes,  nodes,  and  questions  is 
a  set  of  nine  driving  forces,  derived  from  an  original  set  of  more  than 
one  hundred. 
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Also  put  forth  in  the  framework  are  scenarios  of  possible  process 
needs  created  by  combining  the  driving  forces  in  plausible,  but  not 
necessarily  expected,  ways.  Scenarios  provide  unique  narratives  of 
what  the  world  could  look  like  as  various  driving  forces  come  into 
play  and  how  those  forces  could  influence  the  tightly  coupled  con¬ 
nections  between  people,  processes,  projects,  products,  services,  and 
organizations.  Overall,  the  framework  is  meant  to  be  a  comprehensive 
look  at  the  process  field  both  in  its  current  state  and  in  possible  future 
states.  We  hope  that  both  the  detail-oriented  and  big-picture  thinkers 
among  our  readers  will  find  something  to  catch  their  imagination  and 
attention. 

One  of  the  particularly  interesting  attributes  of  collaborative  work,  of 
which  this  framework  is  an  example,  is  access  to  different  voices  and 
experiences  by  which  to  express  ideas.  With  the  process  community 
being  so  diverse — involving  professionals  from  systems,  software, 
services,  acquisition,  development,  operations,  and  maintenance 
across  numerous  industry  and  government  domains — the  perspectives 
of  the  various  members  are  intentionally  left  visible  to  reflect  the  di¬ 
versity  and  breadth  of  the  community. 

In  summary,  this  framework  offers  an  extensive  examination  of  pro¬ 
cess  management  research.  We  hope  that  strategic  planners,  process 
researchers,  and  project  and  product  managers  will  find  it  to  be  a 
helpful  guide.  Yet,  even  this  extensive  body  of  work  is  only  the  first 
step  in  a  long-term  initiative.  The  SEI  plans  to  make  this  framework 
available  in  an  online  community  forum,  enabling  the  growing  base 
of  practitioners,  researchers,  and  other  stakeholders  to  participate 
in  its  evolution.  The  authors  and  editors  of  this  Process  Research 
Framework  hope  that  you  will  find  the  rest  of  this  document  adds 
value  to  your  thinking  and  that  you  will  join  us  in  this  endeavor  as  we 
prepare  to  meet  the  challenges  and  opportunities  ahead  of  us. 

-  Dr.  Caroline  Graettinger,  chair  of  the  IPRC 
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Forces  Driving  Process  Research  Framework 


1-  Value  Add:  the  pushback  from  users  and  customers  who  are 
demanding  more  value-added  solutions 

2.  Business  Diversification:  businesses  changing — getting  larger 
or  smaller,  adopting  different  structures  and  ways  of  working,  or 
entering  new  markets 

3.  Technology  Change:  the  impact  of  new  technologies  both  on 
businesses  and,  particularly,  on  systems  development  operations 

4.  System  Complexity:  new  and  evolving  system  architectures 

5.  Product  Quality:  the  need  to  enhance  product  quality  levels 

6.  Product  Turnaround:  the  speed  and  nature  of  delivery  of 
software-intensive  technological  solutions 

7.  Regulation:  new  and  emerging  standards  that  business  is  or  will 
be  bound  to  follow;  this  could  include  standards  for  the  business 
community  at  large  or,  more  specifically,  standards  imposed  on  the 
development  and  evolution  of  software-intensive  technological 
solutions 

8.  Security  and  Safety:  aspects  of  security  for  the  development 
and  deployment  of  systems  as  well  as  both  the  security  and  safety 
associated  with  the  operation  of  software-intensive  systems 

9-  Globalization:  global  markets,  global  work  forces,  and  cultural 
diversification 
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About  the  IPRC 


In  2004,  the  SEI  formed  the  International  Process  Research  Consortium 
(IPRC)  to  explore  strategic  research  directions  in  software  and  systems 
processes.  The  three  fundamental  components  of  the  IPRC  are  depicted 
in  Figure  1:  (1)  the  strategic  goal,  which  is  essentially  permanent;  (2)  a 
process  research  framework,  embodied  in  this  document  and  future  edi¬ 
tions;  and  (3)  high-priority  process  research  initiatives. 


Figure  1:  Organizational  Components  of  the  IPRC 

Strategic  Goal 

The  strategic  goal  of  the  IPRC  is  to  foster  an  international  community 
of  researchers  and  practitioners  who  collaborate  to  formulate  a  long¬ 
term  process  research  agenda  that  catalyzes  investment.  Strategic  crite¬ 
ria  set  the  tone  for  all  the  work  of  the  IPRC  and  are  the  basis  on  which 
all  future  work  will  be  selected  or  defined.  The  strategic  criteria  are 
these: 

•  developing  an  agenda  that  invites  collaboration  and  stimulates 
innovation 

•  focusing  on  long-term  issues 

•  a  resulting  agenda  of  importance  to  the  international  community 
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Process  Research  Framework 


This  document,  the  process  research  framework,  is  the  first  deliver¬ 
able  of  the  IPRC.  The  purpose  of  the  framework  is  to  (1)  offer  an 
organization  of  the  process  research  needs  for  the  next  1 0  years,  based 
on  the  expertise  and  imagination  of  the  IPRC  members,  and  (2)  act  as 
a  catalyst  for  community-wide  discussions  that  further  enhance  and 
expand  the  framework  for  future  editions.  Current  plans  include  the 
conversion  of  this  document  into  an  online,  interactive  site  where  the 
larger  process  community  will  be  able  to  contribute  to  the  framework. 
The  framework  will  become  a  living  document,  offering  the  process 
community  a  unique  opportunity  to  influence  the  strategic  direction  of 
our  field. 

The  scope  of  this  first  edition  of  the  framework  centers  on  the 
following  focal  question: 

How  should  the  community  represented  by  the 
IPRC  membership  invest  in  process  research 
over  the  next  10 years? 

We  use  this  phrasing  deliberately;  that  is,  we  are  interested  not  only  in 
what  research  should  be  prosecuted,  but  also  how  that  research  may 
be  organized  and  who  might  participate. 

To  answer  our  focal  question  requires  consideration  of  another 
question: 

What  will  be  the  nature  of  software,  systems, 
and  the  enterprise  over  the  next  10 years? 

To  address  this  question,  one  must  consider  what  drives  change 
in  the  first  place: 

What  are  the  drivers  that  will  change  the  nature 
of  software,  systems,  and  the  enterprise  over 
the  next  1 0  years  ? 

These  three  questions  are  central  to  the  ideas  and  recommendations 
contained  in  this  document. 
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Priority  Research  Initiatives 


Using  the  framework  as  a  guide  and  current  events  and  trends  as  lead¬ 
ing  indicators  for  the  future,  every  12-18  months  the  IPRC  will  iden¬ 
tify  high-priority  topics  for  directed  research  initiatives.  The  research 
teams  will  be  composed  of  experts  from  multiple  disciplines,  with  a 
focus  on  stimulating  innovative  thinking  and  new  solutions. 

These  research  initiatives  will  have  to  satisfy  the  following  technical 
criteria  as  well  as  attracting  funding  and  sponsorship: 

•  The  research  is  significant  and  satisfies  the  strategic  goals  of  the 
IPRC. 

•  The  research  is  within  the  scope  of  the  framework. 

•  Current  events  and  trends  suggest  that  the  research  addresses  a 
plausible  future  state. 

Note  that  the  criteria  include  a  focus  on  plausible  future  states.  This 
concept  is  discussed  more  in  Sections  2  and  9  and  is  an  important 
feature  of  the  work  of  the  IPRC. 


Sponsoring  Organizations 
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To  the  Reader 


This  document  is 
meant  to  provide 
information  that  is 
useful  to  several 
different  audiences. 


Of  general  use  to  all  readers  is  the  structured  view  of  process  research 
developed  by  the  group  of  leaders  who  make  up  the  consortium.  In 
addition,  readers  may  choose  to  try  the  techniques  and  structures  we 
have  used  to  organize  their  own  content. 

For  example,  a  reader  from  industry  may  choose  to  apply  particular 
driving  forces  that  are  important  in  their  domain  to  any  and  all  of  the 
research  nodes  to  develop  a  set  of  research  questions  that  are  finely 
tuned  to  their  specific  needs. 

Strategic  planners  may  examine  our  drivers  and  scenarios  and  use 
these  to  validate  or  examine  their  own  assumptions  about  the  future. 

Funding  agencies  may  use  our  themes,  trends,  categories,  and  re¬ 
search  questions  in  evaluating  research  proposals  they  receive. 
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1  Introduction  to  the  Framework 


The  process  research 
framework  includes 
four  research  themes 
and  a  formulation 
of  research  content 
driven  by  emerging 
trends. 


This  introductory  section  explains  what  the  process  research  framework 
contains,  how  it  was  developed,  and  how  it  can  be  used  to  identify 
pertinent  research  for  any  organization.  (Additional  details  on  that  last 
subject  are  found  in  Section  2). 

The  process  research  framework  that  follows  has  two  main  parts.  First 
is  a  set  of  four  research  themes.  Second  is  a  formulation  of  research 
content  driven  by  emerging  trends;  these  may  stand  alone  or  be  relevant 
in  any  or  all  of  the  four  prior  themes. 

The  process  research  topics  represent  an  intersection  of  the  strategic 
goals  of  the  International  Process  Research  Consortium  (IPRC)  and  the 
priorities  of  the  IPRC  members.  The  framework  provides  an  answer  to 
the  IPRC’s  focal  question:  How  should  the  community  represented  by 
the  IPRC  membership  invest  in  process  research  over  the  next  1 0  years? 

Though  the  framework  is  an  answer  to  that  focal  question,  it  is  chock 
full  of  other  questions.  This  is  what  the  framework  provides:  questions, 
not  answers.  It  is  intended  to  provide  guidance  to  industry,  researchers, 
and  funding  agencies  in  determining  the  fruitful  questions  to  address. 
Programs  of  research  or  investigation  can  then  be  formulated  to  answer 
these  questions.  Such  programs  of  research  are,  themselves,  beyond  the 
scope  of  this  document. 

In  August  2004,  the  IPRC  began  the  development  of  what  we  called 
then  a  process  research  roadmap.  To  signal  an  important  difference 
between  our  product  and  typical  roadmaps,  we  now  call  our  document 
a  framework. 

By  its  very  nature,  research  is  highly  iterative.  Further,  the  consortium 
members  aspired  to  develop  a  roadmap  that  was  applicable  in  a  wide 
variety  of  contexts.  One  of  the  challenges  for  the  IPRC  roadmap  archi¬ 
tecture  has  therefore  been  to  formulate  a  structure  that  does  not  presup¬ 
pose  a  particular  order  in  which  research  topics  must  be  addressed,  but 
at  the  same  time  provides  value  by  indicating  linkages  between  areas. 

In  this  way,  we  are  at  odds  with  the  conventional  thinking  on  technol¬ 
ogy  or  product  roadmaps.  A  prominent  feature  of  these  is  to  recommend 
or  predict  a  particular  order  or  sequence  of  development.  As  we  began 
our  work,  we  examined  several  exemplar  roadmaps  and  documents 
about  roadmapping.  Many  of  these  were  product-oriented  or  service- 
oriented  roadmaps  [Wells  2004,  Phaal  2004]  developed  by  organiza¬ 
tions  as  part  of  their  planning  exercises.  The  idea  was  that  when  par¬ 
ticular  events  occurred,  the  product  would  evolve  in  a  specific  direction. 
While  there  may  often  be  an  element  of  uncertainty  in  the  timing,  such 
roadmaps  generally  include  some  notion  of  a  timeline,  or  at  least  an 
ordered  sequence  of  events. 
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As  the  work  of  the  IPRC  developed,  it  became  clear  that  the  same  no¬ 
tion  of  a  sequence  might  not  be  so  easily  accommodated.  The  principal 
distinction  between  a  product-oriented  roadmap  and  what  the  IPRC  is 
interested  in — that  is  a  research  roadmap — is  that  while  some  ordering 
of  activities  might  be  desirable,  often  research  progresses  simultane¬ 
ously  across  a  range  of  areas,  some  of  which  could  be  considered  more 
“advanced”  than  others.  This  is  a  manifestation  of  having  some,  but 
incomplete,  knowledge  of  several  subject  areas.  The  little  bit  of  knowl¬ 
edge  or  progress  in  one  area  may  enable  research  to  begin  in  another 
area — no  need  to  wait  until  the  first  area  is  fully  resolved.  In  addition, 
some  topics  are  more  relevant  in  one  context  than  in  another,  and  we 
wished  to  produce  something  useful  for  the  many  domains  of  practice 
that  depend  upon  software  and  systems. 

The  work  of  the  IPRC  started  with  a  partly  structured  elicitation  of 
ideas  and  background  knowledge  from  this  large  group  of  individuals. 
The  overall  group  then  split  into  two.  One  group  set  about  refining 
and  enlarging  scenarios  for  the  future  from  an  appreciation  of  nine 
emerging  technical  and  business  trends  and  driving  forces  (presented  in 
Section  2).  This  group  was  termed  the  “requirements  pull”  group.  They 
were  trying  to  pull  research  ideas  in  directions  where  we  have  a  less 
well-established  need  for  research  because  of  inattention  to  a  plausible 
scenario  for  the  future  or  uncertainty  about  which  forces  might  prevail. 

The  other  group  started  refining,  categorizing,  and  documenting  known 
process-related  research  activity.  This  group  was  termed  the  “research 
push”  group.  They  were  considering  directions  in  which  current 
research  is  pushing  the  process  community. 

The  combined  work  of  these  two  groups  has  resulted  in  a  core 
framework  that  identifies  areas  of  current  need  for  process  research. 

It  can  also  be  used  in  combination  with  future  trends  to  identify  new 
areas  of  research  need,  perhaps  for  particular  development  environ¬ 
ments  or  a  particular  organization. 

The  principal  constituents  of  the  core  research  framework  are 

•  research  themes 

•  research  nodes 

•  research  questions 
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Step  1:  Identify  problems 
and  drivers  of  change 

Step  2:  Characterize 
future  states 

2a.  Using  known  trends 
2b.  Creating  plausible 
scenarios 


Four  broad  process  research  themes  emerged  by  abstracting  character¬ 
istics  from  the  12  scenarios  created  by  the  requirements-pull  group.  By 
developing  themes  through  a  process  of  combining  forces  into  complex 
scenarios  rather  than  identifying  critical  process  needs  in  response  to 
narrow  technical  or  business  trends,  it  is  hoped  that  the  resultant  frame¬ 
work  will  be  more  widely  applicable  and  resilient  to  a  number  of  possi¬ 
ble  futures.  Much  of  the  research-push  group’s  output  was  also  brought 
to  bear  in  the  resulting  elaborations  of  each  of  these  research  themes, 
through  the  research  nodes  and  associated  research  questions. 


Step  3:  Characterize 
current  capabilities 

3a.  State  of  the  practice 
3b.  State  of  the  art 


Step  4:  Identify  the  gaps 
between  future  states 
and  current  capabilities 


Each  research  theme  embodies  a  common  or  recurring  topic  that  ap¬ 
pears  or  is  implicit  through  several  of  the  IPRC-developed  scenarios. 

A  theme  provides  a  way  of  categorizing  a  set  of  issues  that  have,  at  their 
core,  one  main  focus.  Each  research  theme  is  oriented  to  developing 
process  capabilities  from  the  state  of  today,  to  a  new  state  for  tomorrow, 
whereby  the  process  needs  for  the  likely  business  and  technical  environ¬ 
ment  of  tomorrow  can  be  more  completely  met. 


Step  5:  Formulate 
research  questions  that , 
when  answered,  will  help 
close  the  gaps 


Each  research  theme  is  decomposed  into  a  set  of  research  nodes. 

A  research  node  is  a  collection  of  research  questions.  Each  research 
node  has  an  objective  that  is  intended  to  address  one  aspect  associated 
with  that  theme’s  focus. 


Step  6:  Identify  common 
research  themes 


Finally,  research  questions  are  the  base  constituents  of  the  framework. 
The  research  questions  associated  with  a  research  node  are  intended 
to  form  a  cohesive  grouping.  Depending  upon  the  environment  of  the 
framework  user,  the  answers  to  some  of  these  questions  may  be  rela¬ 
tively  straightforward.  For  other  questions,  the  answers  may  require 
significant  programs  of  research. 


In  all,  the  core  research  framework  consists  of  four  research  themes,  a 
consideration  of  emerging  trends,  20  research  nodes,  and  more  than  230 
research  questions.  The  broad  aspects  covered  across  the  themes  also 
involve  four  fundamental  ingredients  people,  process,  project,  and  prod¬ 
uct  issues.  The  themes  in  the  IPRC  core  research  framework  are  these: 
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1.  The  Relationships  Between  Processes  and  Product 
Qualities 

This  theme  emphasizes  the  product  perspective.  It  is  concerned 
with  understanding  if  and  how  particular  process  characteristics 
can  affect  desired  product  (and  service)  qualities  such  as  security, 
usability,  and  maintainability. 


2.  Process  Engineering 

This  theme  emphasizes  the  process  perspective.  It  is  concerned 
with  how  to  define  and  build  processes  and  understand  their 
performance. 


3.  Managing  Project  Processes 

This  theme  emphasizes  the  project  organizational  perspective. 

It  is  concerned  with  stakeholder  values  that  drive  the  organizational 
structures  used  to  undertake  project  work. 

These  values  may  be  political,  economic,  or  social.  They  may 
be  driven  by  particular  business  domain  concerns,  market 
conditions,  or  the  nature  of  competitive  environments. 


4.  Process  Deployment  and  Use 

This  theme  emphasizes  the  people  perspective.  It  is  concerned 
with  getting  the  right  processes  effectively  deployed  into  suitable 
organizational  structures  so  that  people  are  enabled  most  efficiently 
to  meet  the  needs  of  the  business. 


While  the  framework  represents  an  intersection  of  the  set  of  possible 
research  topics  and  the  priorities  of  the  IPRC  members,  the  members 
believe  that  others  will  find  the  framework  to  be  a  useful  guide  in  defin¬ 
ing  their  process  research  strategies.  For  example,  users  of  this  core  re¬ 
search  framework  may  select,  from  the  set  of  research  questions,  those 
that  are  pertinent  to  their  environment.  Alternatively,  they  may  choose 
to  apply  particular  driving  forces  that  are  important  in  their  domain  to 
any  and  all  of  the  research  nodes  and  then  develop  a  set  of  research 
questions  that  are  finely  tuned  to  their  specific  needs,  thereby  creating 
their  own  instantiation  of  the  research  framework.  In  this  way,  the  IPRC 
core  research  framework  acts  as  a  tool,  supporting  structured  thinking 
in  the  process  domain.  It  does  not  answer  questions.  It  is  intended  to 
help  determine  the  questions  for  which  answers  are  needed. 
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Architecture  of  the 
Research  Themes 

This  section  was  developed  by  Dr.  George  Wilkie, 
framework  architect,  and  introduces  the  structure 
of  the  four  research  themes  defined  over  the  course 


2.1  The  Core  Process  Research  Areas 


Each  research 
theme  enables  an 
examination  of  how 
process  capabilities 
can  be  developed 
from  the  state  of  the 
practice  today  to  a 
new  state  to  succeed 
in  tomorrow's 
environment. 

The  four  broad  process  research  themes  are  presented  in  Figure  2.  In 
the  main,  these  themes  were  inspired  by  abstracting  characteristics 
from  the  12  scenarios  we  created  in  the  requirements-pull  group  (See 
Section  9  for  brief  summaries  of  the  scenarios).  Much  of  the  research- 
push  group’s  output  can  be  found  in  the  resulting  elaboration  of  each 
research  theme,  through  the  research  nodes  and  associated  research 
questions.  Thus,  the  themes  represent  concerns  from  plausible  future 
settings  and  current  conditions. 

By  allowing  us  to  categorize  a  set  of  issues  that  share  a  single  main 
focus,  the  research  theme  lends  cohesiveness  to  the  presentation  of 
a  common  or  recurring  topic  that  appears  in  or  is  implicit  through 
several  of  the  IPRC-developed  scenarios.  Each  of  our  four  research 
themes  is  structured  to  permit  an  examination  of  how  process  capa¬ 
bilities  can  be  developed  from  the  state  of  the  practice  today  to  a  new 
state  to  succeed  in  tomorrow’s  likely  business  and  technical  environ¬ 
ment.  By  developing  themes  through  a  process  of  abstraction  rather 
than  identifying  critical  process  needs  in  response  to  specific  technical 
or  business  trends,  we  intend  this  framework  to  be  widely  applicable. 

In  all,  the  four  research  themes  include  17  research  nodes  that  togeth¬ 
er  total  more  than  175  research  questions.  Additional  nodes  and  ques¬ 
tions  are  included  in  the  section  on  process  effects  of  emerging  tech¬ 
nologies,  bringing  the  total  questions  to  more  than  230.  The  research 
nodes  and  questions  are  meant  to  lead  us  to  process  capabilities  that 
would  move  us  from  the  current  state  to  the  desired  state,  which  we 
describe  for  each  theme.  However,  the  exact  path  to  progress  from 
one  state  to  the  other  is  not  prescribed;  many  paths  are  possible. 

As  shown  in  Figure  2,  the  four  research  themes  and  their  associated 
nodes  are 

The  Relationships  Between  Processes  and  Product  Qualities 
(Section  3) 

Q.  1  Eliciting  and  Specifying  Product  Quality  Requirements 
Q.2  Establishing  Product-Process  Relationships 

Q.3  Modeling  Product  Quality-Process  Relationships 

Q.4  Verification  and  Validation  of  Product  Qualities 

Q.5  Sustaining  Product  Qualities 

Process  Engineering  (Section  4) 

E.  1  Specifying  Processes  Using  Evidence 

E.2  Organizing  Processes  for  Reuse 

E.3  Providing  Process  Engineering  Infrastructure 
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Managing  Project  Processes  (Section  5) 

P.  1  Operating  Under  Autonomous  Control 
P.2  Managing  Through  Centralized  Cooperation 
P.3  Agreeing  Under  Federated  Collaboration 

Process  Deployment  (Section  6) 

D.  1  Establishing  the  Infrastructure  for  Deployment 

D.2  Motivating  Process  Use 

D.3  Effective  Adoption  and  Deployment 

D.3. 1  Formulating  Process  and  Deployment 
Requirements 

D .  3 . 2  Supporting  Effective  Adoption 
D.3. 3  Assessing  Effectiveness 
D.4  Managing  Ongoing  Process  Deployment 

As  Figure  2  shows,  the  research  nodes  in  some  themes  (such  as  The 
Relationships  Between  Processes  and  Product  Qualities)  show  a  pro¬ 
gression  in  terms  of  evolving  knowledge  (a  dashed  line  connector) 
while  within  and  between  other  themes,  the  nodes  show  operational 
progress  (solid  line  connector). 

The  figure  presents  further  elaboration  on  the  cross-theme  research 
node  interconnections.  The  bold  arrows  to  and  from  the  Managing 
Project  Processes  theme  denote  that  it  can  provide  contextual  overlay 
detail  to  be  used  when  building  an  instantiation.  For  example,  you 
may  be  interested  in  understanding  how  to  manage  process  adoption 
(research  Node  D.3. 2)  for  an  organization  managed  through  central¬ 
ized  cooperation  (research  Node  P.2).  Likewise,  any  of  the  research 
nodes  in  each  of  the  theme  Managing  Project  Processes  can  provide  a 
contextual  overlay  for  any  and  all  of  the  research  nodes  within  all  of 
the  other  themes. 
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2.2  Instantiating  the  Core  Process 
Research  Themes 

The  four  core  process  research  themes  can  act  in  concert  as  a  tool  sup¬ 
porting  structured  thinking  in  the  process  domain.  These  research  themes 
do  not  answer  questions;  rather,  they  are  intended  to  help  determine  what 
questions  need  to  be  answered  in  your  environment. 

The  core  process  research  themes  and  nodes  are  necessarily  generic  and 
intended  to  be  used  as  a  starting  point  in  creating  specific  “mappings” 
(derivations  or  instantiations  of  the  core  areas)  based  upon  the  context  of 
your  environment  and  the  driving  forces  affecting  it.  These  instantiations 
are  essentially  lists  of  specific  questions  motivated  by  or  inspired  from 
the  research  questions  within  the  research  nodes  presented  in  Sections 
3-6.  (See  example  instantiation  in  the  Appendix.) 


People  factors  such 
as  education,  culture, 
and  team  building 
are  pervasive  and 
are  covered,  where 
appropriate, 
throughout  the 
themes. 


The  context  for  your  environment  includes  issues  arising  from  or 
pertaining  to  these  fundamental  areas:  people,  process,  project, 
and  product. 

While  there  is  a  complex  interplay  among  all  four  of  these  “Ps,” 
it  may  help  to  think  of  each  in  the  following  terms: 

•  Products 1  harness  technology  and  have  both  external  and 
internal  qualities. 

•  Projects  involve  organizational  structures  and  have  scope. 

•  People  have  motivations,  abilities,  and  culture. 

•  Processes  provide  the  “glue”  to  hold  it  all  together. 


While  product,  project,  and  process  issues  are  the  primary  focus  of 
specific  research  themes  in  the  core  process  research  area  architecture, 
the  people  factors  such  as  education,  culture,  team  building,  leadership, 
communication,  competencies,  roles,  and  motivation  are  very  important 
and  are  covered,  where  appropriate,  throughout  all  the  themes. 


1  In  our  use,  the  term  "products"  includes  the  concept  of  services. 
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People,  process,  project,  and  product  issues  may  all  be  considered  in  any 
research  theme.  However,  people  issues  are  most  prominent  in  Theme  D; 
processes  are  most  prominent  in  Theme  E;  project  issues  are  most  prominent 
in  Theme  P;  and  product  issues  are  most  prominent  in  Theme  Q. 


►  Figure  4:  Example  Relationships  Between  Research  Nodes  in 
Multiple  Themes 
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The  product,  process,  people,  and  project  organizational  perspectives  on 
the  core  process  research  themes  encompass  a  general  appreciation  of 
the  following  nine  driving  forces: 


Forces  Driving  Process  Research  Framework 

1. 

Value  Add:  the  pushback  from  users  and  customers  who  are 
demanding  more  value-added  solutions 

2. 

Business  Diversification:  businesses  changing — getting  larger 
or  smaller,  adopting  different  structures  and  ways  of  working,  or 
entering  new  markets 

3. 

Technology  Change:  the  impact  of  new  technologies  both  on 
businesses  and,  particularly,  on  systems  development  operations 

4. 

System  Complexity:  new  and  evolving  system  architectures 

5. 

Product  Quality:  the  need  to  enhance  product  quality  levels 

6. 

Product  Turnaround:  the  speed  and  nature  of  delivery  of 
software-intensive  technological  solutions 

7 

Regulation:  new  and  emerging  standards  that  business  is  or  will 
be  bound  to  follow;  this  could  include  standards  for  the  business 
community  at  large  or,  more  specifically,  standards  imposed  on  the 
development  and  evolution  of  software-intensive  technological 
solutions 

8. 

Security  and  Safety:  aspects  of  security  for  the  development 
and  deployment  of  systems  as  well  as  both  the  security  and  safety 
associated  with  the  operation  of  software-intensive  systems 

9. 

Globalization:  global  markets,  global  work  forces,  and  cultural 
diversification 

In  developing  your  instantiation  of  research  in  one  of  the  themes,  you 
may  select  from  the  set  of  research  questions  presented  in  the  research 
nodes  in  Sections  3-6  those  that  are  pertinent  to  your  organizational  or 
project  environment.  Alternatively,  you  may  choose  to  apply  particular 
driving  forces  that  are  important  in  your  domain  to  any  and  all  of  the 
research  nodes  in  order  to  develop  a  set  of  research  questions  that  are 
tailored  to  your  specific  needs  (previously  referred  to  as  creating  an 
instantiation).  Figure  2  includes  driving  forces  with  the  core  process 
research  themes. 


IPRC  Framework  |  Architecture  of  the  Research  Themes 


13 


2.3  Overview  of  the  Core  Process  Research 
Themes 

As  Figure  4  illustrates,  each  research  theme  has  been  given  its  own 
thread  through  the  architecture,  with  varying  concentration  of  product, 
process,  people,  and  project  organizational  issues.  Research  nodes  with¬ 
in  each  theme  are  connected  by  lines  to  denote  broadly  how  knowledge 
in  one  node  assists  research  associated  with  the  next  node.  Some  lines 
also  cross  research  themes,  connected  with  lines  to  indicate  where  there 
are  strong  relationships  between  research  nodes.  However,  in  principle, 
there  could  be  important  relationships  between  any  two  research  nodes 
across  the  entire  map,  depending  on  the  reader’s  areas  of  concern. 

In  some  cases,  a  theme  has  a  strong  operational  view  (this  applies  in 
particular  to  the  Process  Engineering  and  Process  Deployment  themes). 
For  such  themes,  traversing  the  nodes  roughly  clockwise  will  help  in 
understanding  the  operational  progression  implied  through  the  connect¬ 
ing  lines. 

For  the  other  themes,  The  Relationships  Between  Process  and  Product 
Qualities  and  Managing  Project  Processes,  there  are  also  connections 
among  the  associated  nodes.  However,  in  these  themes,  the  progression 
of  connected  nodes  suggests  an  evolution  of  knowledge  associated  with 
increasing  complexity  rather  than  any  form  of  an  operational  concept. 

The  Managing  Project  Processes  theme  also  cuts  across  all  three  of  the 
central  themes.  For  this  reason,  it  is  separated  from  the  others. 

A  description  of  each  of  four  core  process  research  themes  follows. 

2.3.1  Research  Theme  Q:  The  Relationships  Between 
Process  and  Product  Qualities 

This  research  theme  provides  the  product  perspective.  It  is  concerned 
with  understanding  if  and  how  particular  process  characteristics  can  af¬ 
fect  desired  product  (and/or  service)  qualities  such  as  security,  usability, 
or  maintainability.  Research  nodes  in  this  theme  can  be  seen  as  project¬ 
ing  an  expected  evolution  of  knowledge,  as  research  is  conducted. 

1 .  The  first  evolutionary  phase  will  move  the  practice  from  an 

understanding  of  how  best  to  define  and  specify  product  qualities 
(the  first  research  node,  Eliciting  and  Specifying  Product  Quality 
Requirements)  to  the  need  to  determine  if  and  how  particular 
processes  and  variations  in  processes  affect  the  level  of  desired 
product  qualities  that  can  be  achieved  (the  second  research  node, 
Establishing  Process — Product  Quality  Relationships). 
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2.  These  advances  will  be  followed  by  the  need  to  understand 
how  to  confirm  achievement  of  quality  requirements  (the  fourth 
research  node,  Verification  and  Validation  of  Product  Qualities). 

3.  Knowledge  gained  from  research  activity  associated  with  these 
three  nodes  can  then  be  used  to  construct  models  of  product- 
quality/process  relationships  which  can  be  employed  as  support 
tools  during  process  selection  (the  third  research  node,  Modeling 
Product  Quality-Process  Relationships). 

4.  The  fifth  research  node  in  this  theme  is  concerned  with  how  to 
select  processes  that  sustain  and  evolve  desired  product  qualities 
throughout  the  full  product  life  cycle  (Sustaining  Product 
Qualities). 

The  relationship  between  an  end-product  quality  and  a  process  is  im¬ 
portant  because  process  selection  and  general  adoption  by  systems  and 
software  engineering  companies  requires  staff  to  understand  the  achiev¬ 
able  benefits.  People  will  only  adopt  and  follow  processes  that  provide 
proven  benefits.  Those  benefits  may  be  in  terms  of  (a)  creating  a  better 
end  product  (or  service),  (b)  making  things  happen  faster,  or  (c)  mak¬ 
ing  people’s  jobs  easier.  Point  (a)  is  covered  throughout  this  research 
theme,  notably  in  Node  Q.2  (Establishing  Process — Product  Quality 
Relationships).  Point  (b)  is  addressed  by  research  Node  E.2  (Organizing 
Processes  for  Reuse)  in  the  Process  Engineering  theme.  Point  (c)  is 
covered  by  research  Node  P.l  (Motivate  Process  Use)  in  the  Process 
Deployment  theme.  Hence,  Figure  3  shows  linkages  among  those  nodes 
and  themes. 

2.3.2  Research  Theme  E:  Process  Engineering 

This  research  theme  provides  the  process  perspective.  It  is  concerned 
with  how  to  define  and  build  processes  and  understand  their  perform¬ 
ance.  The  research  nodes  for  this  theme  consequently  follow  an  opera¬ 
tional  progression  of  how  to 

1 .  specify  processes  with  adequate  empirical  evidence  of  their 
performance  or  performance  impact  in  light  of  requirements 

2.  engineer,  assemble,  combine,  and  reuse  process  components  to 
meet  required  performance  targets 

3.  determine  the  infrastructure  needed  to  best  support  this  process 
engineering  environment 

This  theme  naturally  draws  upon  knowledge  and  understanding  de¬ 
veloped  in  the  first  theme,  since  all  processes  are  enacted  in  support 
of  some  product  or  service.  Likewise,  this  second  theme  draws  upon 
knowledge  and  understanding  from  the  Process  Deployment  theme; 
after  all,  there  is  no  point  in  making  processes  that  people  cannot  or  will 
not  use. 
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2.3.3  Research  Theme  P:  Managing  Project  Processes 


This  theme  provides  the  project  organizational  perspective.  It  is  con¬ 
cerned  with  the  environmental  and  stakeholder  values  that  drive  the  or¬ 
ganizational  structures  used  to  undertake  project  work.  These  values  may 
be  political,  economic,  or  social.  They  may  be  internal  or  external.  They 
may  be  driven  by  particular  business  domain  concerns,  market  condi¬ 
tions,  or  the  nature  of  competitive  environments.  Stakeholders  may  in¬ 
clude  senior  managers,  shareholders,  project  staff,  and  project  customers. 
The  theme  cuts  across  all  three  of  the  other  core  research  themes,  draw¬ 
ing  from  and  having  an  influence  on  all  of  them.  It  is  therefore  presented 
separately  from  the  other  three  in  Figure  2. 

2.3.4  Research  Theme  D:  Process  Deployment 

This  research  theme  strongly  provides  the  people  perspective.2  It  is  con¬ 
cerned  with  getting  the  right  processes  effectively  deployed  into  suitable 
organizational  structures,  so  that  people  are  motivated  and  prepared  most 
efficiently  to  meet  the  goals  of  their  organizations.  The  research  nodes  in 
Process  Deployment  theme  follow  this  operational  progression: 

1.  establishing  the  deployment  infrastructure  (e.g.,  software 
engineering  process  group)  that  will  be  the  focal  point  for 
deployment 

2.  motivation  for  process  use,  selecting  the  appropriate  deployment 
technique,  and  then  planning  for  deployment 

3.  process  deployment  through  formulating  process  requirements, 
adopting  engineered  processes,  and  assessing  process  adoption — 
while  constantly  monitoring  and  controlling  the  ongoing  process 
deployment  activities 

This  theme  must  therefore  interact  with  the  Process  Engineering  theme 
to  ensure  that  appropriate  processes  are  engineered  that  can  be  deployed 
and  will  be  used  and  the  Managing  Project  Processes  theme,  which  is 
concerned  with  project  organizational  issues,  to  ensure  that  process  de¬ 
ployment  suitably  facilitates  the  varying  needs  of  different  organizational 
project  structures. 


2  "People  issues"  are,  of  course,  crucial  and  pervade  all  the  themes,  not  just  this  theme. 
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2.4  How  Each  Theme  Presentation  is 
Structured  in  this  Document 

In  the  presentation  of  each  research  theme,  we  have  included 

•  an  introduction  of  each  research  theme  that  includes  its  rationale  and 
focus 

•  a  description  of  the  state  of  the  practice  today  with  respect  to  the  theme 

If  the  description  we  provide  of  the  current  state  of  any  of  the  core  research 
theme  does  not  match  your  situation,  then  you  can  use  this  document  to 
create  your  own  description  of  where  you  are  today — and  where  you  need 
to  go. 

•  a  description  of  an  idealized  state  of  the  practice  for  tomorrow  with 
respect  to  the  theme 

•  a  description  of  each  research  node  within  the  research  theme 

Each  of  the  identified  research  nodes  within  the  theme  addresses  one  aspect 
of  the  theme’s  focus  and  is  described  in  terms  of  the  process  objectives 
that  practitioners  would  seek  to  achieve.  Singly  or  in  combination  with 
the  objectives  of  other  associated  research  nodes,  each  node’s  objectives 
contribute  to  the  progress  of  the  research  theme.  For  example,  the  focus 
of  the  Process  Deployment  research  theme  is  on  “getting  processes  into 
actual  practice.”  One  of  the  research  nodes  associated  with  this  theme  is 
called  Managing  Ongoing  Process  Deployment,  which  has  the  objective  to 
ensure  that  deployment  of  processes  is  planned,  monitored,  controlled,  and 
appraised  for  effectiveness. 

•  lists  of  research  questions  associated  with  each  research  node 

These  questions,  the  base  constituents  of  the  themes,  are  intended  to  be 
ones  for  which  adequate  answers  are  not  currently  available.  Depending 
upon  your  environment,  however,  you  may  find  answers  to  some  questions 
to  be  relatively  straightforward;  for  many  other  questions,  finding  answers 
will  require  that  you  add  your  context.  For  example,  in  the  research  node 
named  Managing  Ongoing  Process  Deployment,  research  question  D50 
asks:  “How  can  we  best  monitor  the  ongoing  returns  on  investment  on 
process  deployment  and  improvement?  Is  ROI  an  important  measure?” 
These  questions  will  certainly  result  in  a  series  of  related  specific  questions 
posed  in  the  context  of  your  environment,  such  as 

•  How  do  we  demonstrate  that  adopting  a  prototyping  cycle  will  result  in 
significant  cost  savings  during  post- validation  life-cycle  activities? 

•  What  is  the  cost  of  introducing  prototyping  tool  support? 

•  What  is  the  cost  of  training  staff  and  users  on  the  tools  and  techniques 
involved? 

•  For  a  given  size  of  project,  what  is  the  current  cost  of  rework  resulting 
from  validation  failures? 
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3 

Theme 

The  Relationships 
Between  Processes  and 
Product  Qualities 


This  theme  was  developed  by  Julia  Allen  and 
Barbara  Kitchenham,  with  additional  contributions 
from  Mike  Konrad. 


Research  Nodes  and  Questions  in  this  Theme 

•  Research  Node  Q.1 : 

Eliciting  and  Specifying  Product  Quality  Requirements 
(Q1-Q5) 

•  Research  Node  Q.2: 

Establishing  Product — Process  Relationships 
(Q6-Q9) 

•  Research  Node  Q.3: 

Modeling  Product  Quality — Process  Relationships 
(Q10-Q18) 

•  Research  Node  Q.4: 

Verification  and  Validation  of  Product  Qualities 
(Q19-Q23) 

•  Research  Node  Q.5: 

Sustaining  Product  Qualities 
(Q24-Q26) 


3 


3.1 


Introduction  ofTheme 


This  research  theme  is  concerned  with  research  needed  to  understand 
the  relationship  between  process  selection,  project  constraints  (e.g., 
lead  time  or  effort),  and  product3  quality  requirements4  (e.g.,  usabil¬ 
ity  or  security  requirements).  The  responses  to  the  research  questions 
we  present  here  are  most  likely  going  to  ensure  that  product  qualities 
are  effectively  addressed  in  process  definitions  and  their  instantia¬ 
tions — meaning  that  the  products  developed  and  operated  using  specific 
processes  are  able  to  achieve  their  required  product  qualities.  The  scope 
of  these  questions  includes  reaching  an  understanding  of  the  degree  to 
which  processes  support  the  achievement  of  specified  product  quality 
requirements. 

The  theme  addresses  the  following  aspects: 

•  eliciting  and  specifying  product  quality  requirements  in  the  context 
of  process  (research  Node  Q.l) 

•  investigating  the  relationship  between  process  and  quality  in  the 
context  of  different  process  maturity  levels  (research  Node  Q.2) 

•  modeling  product  quality  and  process  relationships 
(research  Node  Q.3) 

•  selecting  a  process  to  deliver  specific  product  qualities 
(research  Node  Q.3) 

•  assessing  the  extent  or  degree  to  which  the  product  quality  is  present 
(research  Node  Q.4) 

•  verifying  and  validating  product  qualities  (research  Node  Q.4) 

•  making  a  product  quality  tradeoff  analysis  (including  tradeoffs 
between  product  qualities  and  business  requirements  such  as  cost 
and  schedule)  (research  Node  Q.3) 

•  sustaining  product  qualities  in  operation  and  product  evolution 
(research  Node  Q.5) 

This  theme  description  is  generic;  that  is,  our  discussion  and  research 
questions  are  meant  to  be  applicable  to  most  product  qualities.  Research 
questions  can  be  specialized  to  address  issues  related  to  specific  product 
qualities;  see  the  Appendix  for  an  example  of  research  questions  spe¬ 
cialized  for  “security.”  Specific  product  qualities  that  may  be  of  interest 
include  those  below.  They  are  not  an  exhaustive  list,  but  are  those  that 
arose  often  in  our  deliberations. 


3  In  this  description,  product  is  synonymous  with  systems  of  systems,  system,  and  soft¬ 
ware. 

4  The  term  product  qualities  refers  to  nonfunctional  requirements  such  as  security  as  well 
as  behavioral  requirements  such  as  performance  or  reliability. 
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•  security 

•  usability  (human-machine  interfaces) 

•  reliability 

•  dependability 

•  safety 

•  interoperability 

•  maintainability 

With  respect  to  tradeoffs  and  processes  specifically  intended  to  accom¬ 
plish  improvements  in  cost  and  schedule  (such  as  processes  for  software 
product  lines),  product  qualities  that  embody  these  include  affordability 
and  timeliness.  These  qualities  need  to  be  included  within  the  scope  of 
all  product  qualities  when  considering  the  research  nodes,  objectives, 
and  research  questions  that  follow. 

3.2  Theme  Rationale 

This  research  theme  is  concerned  with  understanding  process  character¬ 
istics  that  address  product  qualities.  The  motivation  behind  this  theme  is 
one  of  demonstrating  a  more  direct  linkage  between  processes  and  end- 
product  characteristics. 

Currently,  we  see  no  general  agreement  about  the  relationship  between 
process  and  product  quality.  The  lack  of  agreement  is  visible  in  the 
variety  of  different  approaches  to  product  quality  adopted  by  different 
researchers  and  research  groups: 

•  ISO/IEC  9126  takes  a  “product”  view.  It  provides  a  hierarchical 
model  based  on  defining  six  high-level  quality  characteristics 
(functionality,  efficiency,  portability,  maintainability,  reliability, 
usability)  and  decomposing  them  into  subcharacteristics  and  finally 
into  measurable  attributes.  This  approach  has  many  problems,  not 
the  least  of  which  is  the  inconsistencies  in  the  standard  itself  [Al- 
Kilidar  2005]. 

•  Software  metrics  researchers  concentrate  on  internal  quality 
assessment  and  investigate  measures  of  “complexity,”  “coupling,” 
and  “cohesion”  of  software  components  [El-Emam  1999].  ISO  9126 
is  incompatible  with  this  approach. 

•  Reliability  modeling  and  performance  modeling  research  concentrate 
on  the  properties  of  a  system  in  its  operational  environment  (or 
simulated  operational  environment).  This  research  uses  definitions 
of  quality  that  differ  substantially  from  those  of  ISO  9126  and  are 
more  suitable  for  requirements  specification.  Reliability  research  is 
part  of  a  wider  research  community  interested  in  the  dependability 
of  software  systems  where  dependability  includes  concepts  such  as 
availability,  reliability,  safety,  and  security. 
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Current  research  results 
make  little  attempt  to 
provide  an  integrated 
approach  to  product 
quality  requirements. 

•  Usability  design  and  testing  (via  usability  laboratories)  takes 
a  different  approach  to  usability  than  ISO  9126,  referencing 
completely  different  standards: 

•  ISO  9241:  Ergonomic  requirements  for  office  work  with  visual 
display  terminals  (VDTs)  -  Part  1 1 :  Guidance  on  usability 

•  ISO/TR  16982:  Ergonomics  of  human- system  interaction; 
usability  methods  supporting  human-centered  design 

•  ISO  13407:  Human-centered  design  processes  for  interactive 
systems 

•  Current  software  architecture  research  considers  quality  from 
the  viewpoint  of  assessing  appropriate  architectures  for  new 
products.  One  approach  is  to  develop  scenarios  that  define  quality 
requirements  and  (design  and  architectural)  patterns  that  help  to 
achieve  these  requirements.  Stakeholders  are  expected  to  define 
quality  requirements  in  terms  of  concrete  scenarios  defining  system 
behavior  or  use.  However,  until  recently,  the  assessment  of  the 
quality  levels  that  will  be  achieved  by  design  patterns  has  been 
qualitative  (e.g.,  a  specific  pattern  might  “increase”  reliability  and 
“slightly  decrease”  maintainability).  See,  for  example,  Buschmann  et 
al.  on  pattern-oriented  software  architecture  [Buschmann  1996]  and 
Bass  et  al.  Software  Architecture  in  Practice  [Bass  2003]. 

These  approaches  use  inconsistent  terminology  and  incompatible 
methods.  ISO  9126  defines  each  quality  as  “a  set  of  attributes  that 
bear  on. . .”  (For  instance,  “Reliability  is  a  set  of  attributes  that  bear  on 
the  capability  of  software  to  maintain  its  level  of  performance  under 
stated  conditions  for  a  stated  period  of  time.”)  The  dependability  defini¬ 
tions  define  the  quality  directly  (e.g.,  reliability  is  “continuity  of  correct 
service”).  (Note  that  this  definition  is  often  operationalized  as  “the  prob¬ 
ability  of  failure-free  software  operation  for  a  specified  period  of  time 
in  a  specified  environment”).  There  is  a  substantial  difference  between 
regarding  reliability  as  a  set  of  attributes  and  regarding  it  as  a  property 
of  software  in  its  own  right. 

Victor  Basili  suggests  viewing  quality  characteristics  not  by  name  but 
by  user  issue  (e.g.,  what  failure  will  I  not  abide,  what  hazards  are  unac¬ 
ceptable).  His  view  eliminates  the  need  for  a  particular  product  quality 
definition  and  aims  at  the  needs  of  the  user.  We  may  not  need  to  define 
a  quality  but  only  specify  what  is  not  allowable.  For  views  on  depend¬ 
ability,  see  recent  articles  by  Basili  and  Donzelli  [Basili  2004,  Donzelli 
2005,  Donzelli  2006].  For  another  perspective  on  defining  product  qual¬ 
ities,  see  Voas’  “Software’s  Secret  Sauce:  The  ‘-ilities’”  [Voas  2004]. 

Current  research  results  make  little  or  no  attempt  to  provide  managers 
and  engineers  with  an  integrated  approach  to  specifying,  achieving,  and 
validating  product  quality  requirements.  We  need  a  coherent  research 
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plan  to  address  the  underlying  issues  facing  software  producers  (i.e., 
how  to  design  their  development  processes  and  manage  their  software 
projects  in  order  to  achieve  specified  product  quality  requirements). 

3.3  Characterizing  the  Current  State  of  the 
Practice 

In  this  section,  we  describe  the  current  state  of  the  practice  with  respect 
to  product  qualities,  their  presence  or  absence  at  desired  levels  in  devel¬ 
oped  and  deployed  systems,  and  the  extent  to  which  process  is  viewed 
as  necessary  to  achieve  a  given  product  quality. 

Until  recently,  little  strong  evidence  has  linked  software  processes  to 
quantitative  quality  achievements.  For  example,  a  fairly  substantial 
amount  of  research  has  linked  module  or  component  fault  rates  to  in¬ 
ternal  software  component  measures.  However,  these  studies  have  not 
related  module  fault  rates  to  reliability.  Furthermore,  the  models  only 
seem  (even  moderately)  useful  in  the  context  of  evolving  systems  (i.e., 
predicting  fault  rates  of  modules  across  consecutive  releases  of  a  spe¬ 
cific  system  [Ohlsson  2001]).  Attempts  to  measure  internal  properties 
of  a  system  during  software  development — measurements  that  can  be 
used  as  surrogates  for  software  product  qualities  as  proposed  by  ISO 
9126 — have  proven  to  be  unsuccessful. 

Since  the  1970s,  considerable  research  has  been  done  in  the  area  of  reli¬ 
ability  and  performance  modeling.  This  research  has  been  successful 
in  modeling  self-standing  software  systems  at  the  end  of  development 
(i.e.,  during  system  testing  and  acceptance  testing)  if  the  operational 
profile  of  the  system  is  well  understood. 

Recently,  a  good  deal  of  research  has  investigated  the  quality  impacts 
of  architectural  decisions.  Although  much  of  this  research  is  essentially 
qualitative  [Svahnberg  2003],  some  major  advances  have  been  made  in 
quantitative  prediction  of  both  performance  [Liu  2005]  and  reliability 
[Wang  1999]  during  architectural  design  in  the  context  of  component- 
based  system  development.  However,  tradeoff  models,  although  sophis¬ 
ticated,  are  still  qualitative  [Al-Naeem  2005]. 

Some  qualities  map  readily  to  measurable  concepts;  others  do  not. 
Properties  such  as  safety  and  security  are  usually  decomposed  into  non¬ 
functional  requirements;  they  are  not  considered  “measurable”  proper¬ 
ties.  This  circumstance  makes  it  difficult  to  assess  the  level  of  safety  or 
security  achieved  other  than  qualitatively,  even  after  system  delivery. 
For  example,  safety  cases  are  assessed  by  argumentation,  not  measure¬ 
ment  [Weaver  2005]. 
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In  an  ideal  future  state, 
the  use  of  processes 
is  part  of  accepted 
practice  to  ensure  that 
acceptable  levels  of 
product  qualities  are  in 
place  during  all  stages 
of  the  software  and 
system  development 
life  cycle. 

Overall,  progress  in  predicting  quality  characteristics  (and  thus  validat¬ 
ing  quality  requirements)  early  in  the  development  process  is  mixed. 

An  approach  based  on  measuring  internal  software  metrics  was  first 
proposed  in  the  1970s  [Boehm  1978]  and  has  been  the  object  of  many 
research  projects  (e.g.,  the  method  reported  by  Boegh  et  al.  [Boegh 
1999]).  However,  no  effort  based  on  measuring  internal  software  met¬ 
rics  has  delivered  any  major  benefits.  It  is  time  to  recognize  that  this 
approach  does  not  work.  Recent  advances  in  quantitative  performance 
and  reliability  prediction  are  extremely  promising.  They  confirm  the 
value  of  reusing  architectures  and  component-based  software  develop¬ 
ment.  However,  other  characteristics  such  as  usability,  maintainability, 
security,  and  safety  can  only  be  measured  and  predicted  qualitatively. 
Tradeoffs  between  qualities  are  only  possible  using  qualitative  models. 
The  challenge  remains  to  construct  quantitative  tradeoff  models  incor¬ 
porating  most  (if  not  all)  qualities  along  with  cost  (affordability)  and 
schedule  (timeliness),  as  does  the  need  to  model  and  predict  quality 
characteristics  of  complex  systems  of  systems. 

3.4  Characterizing  the  Desired  State  of  the 
Practice 

In  this  section,  we  describe  the  desired  state  of  practice  with  respect  to 
product  qualities  when  all  questions  posed  in  this  core  roadmap  seg¬ 
ment  have  been  satisfactorily  addressed  and  solutions  adopted. 

In  this  ideal  future  state,  processes  are  accepted  as  one  means  by  which 
specified  levels  of  product  qualities  can  be  demonstrated  and  achieved. 
The  use  of  processes  is  part  of  accepted  practice  to  ensure  that  accept¬ 
able  levels  of  product  qualities  are  in  place  during  all  stages  of  the  soft¬ 
ware  and  system  development  life  cycle.  The  relationship  between  pro¬ 
cess  and  specific  product  qualities  is  known  and  can  be  quantitatively  or 
qualitatively  demonstrated  (based  on  the  nature  of  the  product  quality). 

Further,  processes  are  effectively  used  to  elicit  and  capture  product 
quality  requirements  including  their  specified  levels  for  given  products, 
markets,  types  of  systems,  and  the  like.  The  process  disciplines  and 
methods  are  sufficiently  mature  to  provide  confidence  that  expressing 
product  quality  requirements  using  process  methods  will  produce  the 
desired  result  in  the  deployed  product. 

Processes  are  also  effectively  used  to  assist  in  analyzing  and  making 
informed  tradeoff  decisions  among  all  product  quality  attributes  and  be¬ 
tween  given  product  qualities  and  business  requirements  (such  as  cost, 
schedule,  team  competence,  risk).  Some  business  requirements  can  be 
considered  as  product  qualities,  such  as  affordability  as  an  expression 
of  cost  and  timeliness  as  an  expression  of  schedule.  Modeling  methods 
exist  for  guiding  these  tradeoff  analyses. 
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In  addition,  processes  are  effectively  used  to  test,  verify,  and  validate 
that  product  quality  acceptance  criteria  and  acceptance  cases  are  met. 
Processes  exist  to  assess  and  audit  systems  for  aiding  in  making  this  de¬ 
termination.  Measurement  processes  and  defined  measures  are  regularly 
used  to  demonstrate  that  product  qualities  continue  to  be  present  at  the 
required  level  throughout  the  product  life  cycle,  and  in  the  face  of  product 
changes.  (See  Bollinger,  Voas,  and  Boasson  on  persistent  software  attri¬ 
butes  [Bollinger  2004].) 

All  categories  of  process  described  in  this  section  have  automated  support 
to  ensure  their  continued  use  and  ease  of  use.  In  the  state  of  the  practice 
tomorrow,  we  fully  expect  that  the  relationship  of  process  to  some  prod¬ 
uct  qualities  will  be  more  mature  than  for  others. 

3.5  Description  of  the  Research  Nodes 

3.5.1  Research  Node  Q.1:  Eliciting  and  Specifying 
Product  Quality  Requirements 

The  objective  of  this  research  node  is  to  determine  how  best  to  elicit 
quality  requirements  (criteria)  for  a  product  and  how  to  properly  specify 
these. 


Research  questions  associated  with  this  node  include5 


Q-1 

How  can  a  consumer  with  limited  expertise  be  facilitated  to 
express  their  needs  in  a  sufficiently  prescriptive  way? 

Q-2 

How  do  we  specify  product  quality  requirements  in  general  and 
such  that  they  reflect  business  requirements? 

Q-3 

What  levels  of  product  quality  are  required  for  specific  types  of 
products?  For  specific  markets?  How  are  levels  expressed? 

Q-4 

How  do  we  specify  product  quality  requirements  in  a  quantifiable, 
measurable  manner?  (How  will  we  know  it  when  we  see  it?) 

Q-5 

Does  the  level  of  need  for  each  '-ility'  vary  by  business  domain? 

If  so,  how? 

5  We've  given  each  research  theme  a  letter  identifier  so  that  the  questions  associated  with 
each  theme  can  be  more  readily  identified.  We've  identified  this  theme  on  the  relationships 
between  processes  and  product  qualities  with  the  letter  Q.  Other  themes  on  process  en¬ 
gineering  (identified  with  E),  managing  project  processes  (P),  and  process  deployment  (D) 
feature  the  same  treatment.  In  addition,  questions  associated  with  the  effects  of  emerging 
trends  are  identified  with  a  T  and  those  in  the  example  instantiation  for  security  with  an  S. 
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3.5.2  Research  Node  Q.2: 

Establishing  Product-Process  Relationships 

The  objective  of  this  research  node  is  to  establish  whether  there  is  a 
direct  relationship  between  a  given  product  quality  and  the  processes 
used  to  develop  the  product. 


Research  questions  associated  with  this  node  include 


Q-6 

Is  there  a  direct  relationship  between  desired  product  qualities 
and  the  processes  used  to  develop  the  product?  If  so,  how  is 
this  relationship  expressed  and  how  does  it  differ  based  on  the 
maturity  of  the  process  used? 

Q-7 

How  do  we  determine  the  relationship  between  process  and 
product  qualities? 

Q-8 

What  is  the  evidence  that  better  processes  are  instrumental  to 
delivering  better  products? 

Q-9 

Do  the  issues  for  product  quality  and  process  relationships  change 
for  composable  components  and  for  products  assembled  from 
composable  components? 

3.5.3 

Research  Node  Q.3:  Modeling  Product  Quality- 
Process  Relationships 

The  objective  of  this  research  node  is  to  enable  managers  to  select  pro¬ 
cesses  that  result  in  the  desired  product  qualities  at  the  required  level 
and  degree  given  the  maturity  level  of  their  organization.  They  can 
make  informed  tradeoffs  between  product  qualities  and  other  project 
and  business  requirements  as  well  as  among  product  qualities.  They 
understand  the  risks  and  uncertainties  inherent  in  their  choices. 

This  research  node  also  supports  (re)negotiating  product  quality  re¬ 
quirements  in  terms  of  what  is  actually  possible,  given  constraints 
throughout  the  project  and  product  life  cycle. 

Research  questions  associated  with  this  node  include 

Q-10 

How  do  we  model  and  predict  product  quality — process 
relationships  in  the  context  of  different  maturity  levels? 

Q-11 

How  do  we  model  and  identify  the  tradeoffs  between  business 
requirements,  process,  and  product  qualities  (such  as  degree 
of  quality,  cost  (affordability),  schedule  (timeliness),  team 
competence,  risk)? 

Q-12 

How  do  we  model  and  identify  tradeoffs  among  required  product 
qualities? 
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Q-13 

How  do  we  make  well-informed  decisions  using  the  results  of 
trade-off  analyses? 

Q-14 

How  do  we  select  processes  to  meet  specific  product  quality 
requirements? 

Q-15 

What  process  steps  significantly  influence  the  achievement  of  a 
specified  level  of  product  quality? 

Q-16 

When  composing  systems  of  individually  certified  components, 
how  do  you  ensure  that  the  non-functional  product  quality 
attributes  such  as  security  are  achieved  in  the  composed 
system? 

Q-17 

Can  quality  attributes  associated  with  intermediate  work 
products  be  used  as  indicators  for  quality  attributes  in  the  final 
work  product? 

Q-18 

If  not  in  absolute  terms,  can  intermediate  work  products  be  used 
to  inform  the  risks  of  failing  to  achieve  final  product  qualities? 

3.5.4  Research  Node  Q.4:  Verification  and  Validation  of 
Product  Qualities 

The  objective  of  this  research  node  is  to  enable  managers  to  select  appropri¬ 
ate  verification  and  validation  processes  to  confirm  the  achievement  of  quality 
requirements.  Process  selection  is  guided  by  the  nature  and  complexity  of  the 
system  being  constructed  and  operated. 


Research  questions  associated  with  this  node  include 


Q-19 

How  do  we  determine  and  specify  product  quality  acceptance 
criteria? 

Q-20 

How  do  we  test,  verify,  validate,  assess,  and  audit  that  product 
quality  requirements  are  met  in  the  product? 

Q-21 

Using  product  quality  and  process  measures,  how  can  we 
determine  that  we  are  on  track  to  achieve  the  desired  level  of 

product  quality  during  each  life-cycle  phase? 

Q-22 

Do  product  quality  verification  and  validation  approaches 
scale  up  and/or  change  in  the  context  of  different  product 
architectures  (e.g.  complex  systems  and  systems  of  systems)? 

Q-23 

What  is  the  role  of  automation  in  product  quality  verification  and 
validation? 
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3.5.5  Research  Node  Q.5: 

Sustaining  Product  Qualities 

The  objective  of  this  research  node  is  to  enable  managers  to  select 
processes  that  best  facilitate  establishing,  sustaining,  and  evolving 
product  qualities  throughout  the  full  product  life  cycle. 


Research  questions  associated  with  this  node  include 


Q-24 

How  do  we  sustain  product  qualities  in  the  face  of  product 
operations,  product  requirements  change  (both  functional  and  non¬ 
functional),  and  product  evolution? 

Q-25 

How  do  we  measure  (assess,  audit)  that  product  qualities  continue 
to  be  present  at  the  required  level  and  degree  throughout  the 
product  life  cycle? 

Q-26 

What  is  the  role  of  automation  in  product  quality  sustainment? 
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Research  Nodes  and  Questions  in  this  Theme 


Specifying  Processes  Using  Evidence 
(E1-E21) 


•  Research  Node  E.2: 

Organizing  Processes  for  Reuse 
(E22-E43) 


Providing  Process  Engineering  Infrastructure 
(E44-E60) 


4 

Theme  E:  Process 
Engineering 


This  theme  was  originally  developed  by  Dieter  Rombach, 
Ross  Jeffrey,  Bill  Peterson,  Michael  D'Ambrosa,  Mario 
Fusani,  Ho-Won  Jung,  and  Stefan  Ferber.  Jurgen  Munch 
and  Alexis  Ocampo  made  additional  contributions. 


•  Research  Node  E.1 


•  Research  Node  E.3 
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4.1  Introduction  ofTheme 


This  theme  primarily  looks  at  processes  from  a  process  engineering 
perspective.  Incorporated  in  this  theme  is  the  allied  topic  of  integrating 
systems  development  processes  with  domain-specific  processes.  The 
focus  of  the  previous  theme  (relationships  between  process  and  product 
quality)  is  on  product  qualities,  their  definition,  and  the  characterization 
of  processes  that  promote  particular  product  qualities.  The  focus  of  this 
theme  is  on  processes  per  se — how  to  define  a  process,  construct  a  pro¬ 
cess  from  process  components,  and  then  understand  and  model  process 
performance  with  a  view  toward  building  predictive  capabilities  based 
on  experimenting  with  process  models. 

4.2  Theme  Rationale 

This  theme  is  concerned  with  providing  a  set  of  process  artifacts  and 
mechanisms  for  effective  (re)use  by  activities  described  in  the  process 
deployment  theme.  This  objective  includes 

•  specifying  processes  using  evidence  (e.g.,  defining  scope  of 
process,  deriving  process  goals  from  business  needs,  technology, 
socio-political  environment,  and  domain;  providing  quantitative 
process  or  product  relationships  for  relevant  goals)  based  on  results 
from  “relationships  between  processes  and  product  quality”  and 
“deployment  and  use” 

•  organizing  processes  for  (re)use  (e.g.,  identifying  process  families, 
engineering  processes  from  process  components)  with  process  lines 
as  an  optional  architecture  for  process  components 

•  providing  process  engineering  infrastructure  for  selection, 
integration,  tailoring,  and  learning  (e.g.,  providing  methods  and 
tools,  applying  existing  process  frameworks) 

4.3  Characterizing  the  Current  State 
of  the  Practice 

The  current  state  of  process  engineering  is  based  upon  ad  hoc  selection, 
tailoring,  and  integration  without  credible  evidence  of  process  impact. 
While  companies  with  processes  operating  at  CMMI6®  framework  level 
3  or  above  do  have  organizational  standard7  processes  they  tailor  to 


6  ®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

7  This  meaning  uses  the  CMMI  definition  of  a  "standard"  process:  "A  standard  process 
describes  the  fundamental  process  elements  that  are  expected  to  be  incorporated  into 
any  defined  process.  It  also  describes  the  relationships  (e.g.,  ordering  and  interfaces) 
among  these  process  elements." 
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create  defined8  processes  for  specific  projects,  we  know  of  little  com¬ 
monly  agreed  upon  evidence  to  support  the  “goodness”  of  these  pro¬ 
cesses. 

We  also  have  no  commonly  acknowledged  way  to  build  processes 
from  process  components  and  no  clear  understanding  of  how  to  define 
process  component  interfaces  or  to  assess  the  compatibility  of  linked 
process  components.  Processes  are  constructed  often  by  crafting  than  by 
applying  engineering  principles. 

In  summary,  with  the  current  state  of  knowledge,  we  are  unable  to 

•  specify  accurately  the  elements  of  a  process  having  desired 
characteristics  (functionality  and  capability  or  repeatability) 

•  confirm  that  a  process  model,  if  correctly  implemented,  will  meet  the 
requirements  defined  for  its  purpose  (including  business  objectives) 

•  confirm  the  fidelity  and  suitability  of  a  process  from  an  analysis  of 
its  specification 

•  determine  a  strategy  for  constructing  a  process  model  from  known  or 
proven  elements  (subprocesses) 


Today,  we  have 
no  commonly 
acknowledged  way 
to  build  processes 
from  process 
components  and  no 
clear  understanding 
of  how  to  define 
process  component 
interfaces. 


4.4  Characterizing  the  Desired  State 
of  the  Practice 

Process  line  engineering  combines  process  engineering  with  the  concept 
of  product  line  engineering.  Variant-rich  processes  contain  process  ele¬ 
ments  (e.g.,  activities,  inputs,  outputs,  or  roles).  Process  elements  that 
contain  variation  points  are  called  variant-rich  process  elements.  They 
are  organized  in  a  process  line  infrastructure  that  is  designed  with  stra¬ 
tegic  business  goals  in  mind.  Concrete  processes  are  derived  from  the 
process  line  infrastructure  (i.e.,  instantiating  the  process  line  and  resolv¬ 
ing  the  variation  points  contained  in  variant-rich  process  elements). 
Intelligent  support  increases  the  efficiency  of  evidence  collection,  pro¬ 
cess  selection,  tailoring,  and  integration. 


8  Again,  we're  using  the  CMMI  definition  of  a  "defined"  process:  "A  managed  process 
that  is  tailored  from  the  organization's  set  of  standard  processes  according  to  the 
organization's  tailoring  guidelines;  has  a  maintained  process  description;  and  contributes 
work  products,  measures,  and  other  process  improvement  information  to  the  organiza¬ 
tional  process  assets." 
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4.5  Description  of  the  Research  Nodes 


This  theme  has  three  research  nodes: 

•  specifying  processes  using  evidence  (E.  1) 

•  organizing  processes  for  reuse  (E.2) 

•  providing  process  engineering  infrastructure  (E.3) 

The  focus  of  this  theme  is  on  those  processes  for  developing  software¬ 
intensive  systems  or  pure  software  systems. 

4.5.1  Research  Node  E.1: 

Specifying  Processes  Using  Evidence 

The  objective  of  this  research  node  is  to  provide  the  abilities  to  specify, 
select,  and  evaluate  processes  based  on  evidence. 

The  first  step  in  process  specification  is  to  identify  needs  for  credible 
evidence.  This  step  draws  upon  business  goals,  as  well  as  linkages  to 
evidence  from  process  and  product  quality  relationships  and  motiva¬ 
tions  from  process  deployment  and  use.  The  first  of  these  linkages  pro¬ 
vides  evidence  of  the  applicability  of  a  process  to  the  needs  for  one  or 
more  specific  product  qualities.  The  second  linkage  provides  feedback 
on  process  deployment  and  use  and  considerations  of  people  competen¬ 
cy  issues.  This  research  node,  specifying  processes  using  evidence,  then 
integrates  these  sources  of  evidence  and  understanding  so  that  we  can 
reason  clearly  about  the  requirements  or  selection  criteria  for  processes. 

Research  questions  associated  with  this  node  include9 


Scope  of  Specification 


E-1 

How  can  usable  best  practices  be  identified? 

E-2 

What  kinds  of  processes  are  needed  for  value-creating  networks; 
virtual  teams;  partnering;  outsourcing,  multi-site  development,  end- 
user  development? 

E-3 

How  can  we  align  processes  with  business  goals? 

E-4 

How  to  perform  a  gap  analysis  between  today's  state  and  a  desired 
future  state? 

9  We've  given  each  research  theme  a  letter  identifier  so  that  the  quesitons  associated 
with  each  theme  can  be  more  readily  identified.  We've  identified  this  theme  on  process 
engineering  with  the  letter  E.  Other  themes  on  the  relationships  between  processes 
and  product  qualities  (identified  with  Q),  managing  project  processes  (P),  and  process 
deployment  (D)  feature  the  same  treatment.  In  addition,  questions  associated  with  the 
effects  of  emerging  trends  are  identified  with  a  T  and  those  in  the  example  instantiation 
for  security  with  an  S. 
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Mechanisms  for  Specification 

E-5  How  can  we  best  specify  a  process? 


4 


E-6 

How  can  process  definitions  be  packaged  together  with  a 
quantitative/qualitative  model  describing  their  behavior? 

E-7 

What  are  appropriate  process  notations? 

E-8 

Can  a  process  be  analyzed  to  determine  if  it  is  implementable? 

E-9 

What  process  evidence  is  required? 

E-10 

What  evidence  is  required  with  respect  to  process  risks? 

E-11 

How  can  this  evidence  be  specified  and  applied  to  the  selection, 
tailoring  and  integration  of  processes? 

E-12 

How  to  combine  evidence  and  "context"? 

E-13 

How  does  the  context  of  a  process  (e.g.,  organization  size,  culture, 
process  distribution)  influence  process  selection  criteria? 

E-14 

How  can  acquired  process  components  be  evaluated  and  certified 
(so  that  they  can  be  trusted)? 

E-15 

How  can  knowledge  from  the  related  areas  of  organizational 
and  behavioral  studies  be  incorporated  into  the  definition  and 
specification  of  processes  that  can  be  effectively  implemented? 

E-16 

How  can  we  assure  that  a  process  will  meet  product/project 
requirements  and  standard  compliance? 

E-17 

What  does  it  mean  to  certify  a  process  component  and  how 
could  this  be  achieved?  (For  example,  what  criteria  could  such 
certification  be  made  against?) 
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Process  Specification  Improvement 


E-18  How  can  we  improve  process  specifications  based  on  feedback 
from  deployment  and  use? 


E-19  How  can  the  value  of  a  process  be  determined  and  monitored? 


E-20  How  can  the  quality  and  cycle  time  performance  implications  of 
process  decisions  be  evaluated? 


E-21  What  effects  do  different  domains  have  on  the  selection  criteria 
for  processes?  The  critical  issue  here  is  the  need  for  a  clearer 
schema  for  classifying  and  categorizing  "domains."  There  is 
confusion  between  business  domains,  application  domains,  and 
industry  domains,  as  well  as  with  factors  incorporating  cultural 
issues.  Identification  of  critical  domain  characteristics  is  crucial. 
Once  these  are  known,  the  process  and  systems  engineering 
issues  can  be  addressed. 


4.5.2  Research  Node  E.2: 

Organizing  Processes  for  Reuse 

The  objective  of  this  research  node  is  to  engineer  processes  from  reus¬ 
able  process  components  and  other  artifacts  to  meet  specified  process 
requirements.  One  approach  is  to  frame  the  issue  in  the  context  of  pro¬ 
cess  lines. 

Leveraging  the  business  investment  in  processes  for  reuse  in  multiple 
projects,  business  units,  and  sites  is  a  critical  issue  for  process  engineer¬ 
ing.  The  multiple  mechanisms  to  support  reuse  include  process  libraries, 
process  tailoring,  process  standards,  best  practices,  and  expert  experi¬ 
ence-exchange  networks.  Software  product  lines  target  at  strategic  reuse 
in  products  linked  to  the  business  advantage  instead  of  reuse  as  a  goal 
by  itself.  Transferring  the  concept  to  process  engineering  is  the  key  idea 
behind  process  line  engineering: 

•  Generic  process  elements  containing  commonality  and  variability 
are  organized  in  a  process  architecture  that  is  designed  with  strategic 
business  goals  in  mind. 

•  Concrete  processes  are  derived  from  a  process  architecture, 
instantiating  the  process  architecture  and  binding  variation  points  of 
generic  process  elements. 

Typical  variation  points  of  generic  process  elements  are  the  application 
domain,  project  goals,  and  team  competence.  Tailoring  of  processes  in  a 
process  line  becomes  mostly  a  variant  selection  activity.  The  process  ar¬ 
chitecture  and  process  component  integration  are  the  critical  mechanisms 
to  guarantee  that  tailored  processes  fit  together  and  form  a  smoothly  op¬ 
erational  process  landscape.  Finally,  the  reusable  process  components  of 
a  process  line  require  ongoing  improvement  and  evolution. 
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Research  questions  associated  with  this  node  include 

Process  Lines 


4 


E-22 

How  can  we  define  the  scope  of  process  lines? 

E-23 

How  can  we  define  the  value  of  process  lines? 

E-24 

How  can  we  organize  processes  and  evidence  into  one  or  more 
process  lines  (similar  to  the  concept  of  product  lines)?  This 
includes  domain-specific  issues. 

E-25 

What  is  the  appropriate  degree  of  commonality  of  processes/ 
procedures  (e.g.,  across  multiple  sites,  disciplines,  and  cultures)? 

E-26 

How  to  understand  the  difference  between  different  domains  with 
respect  to  processes  and  measurement? 

E-27 

What  effects  do  different  domains  have  on  the  selection  criteria 
for  processes? 

E-28 

How  can  process  line  engineering  be  aligned  with  product  line 
engineering? 

E-29 

How  should  a  process  architecture  be  constructed  for  a  process 
asset  base? 

E-30 

How  can  the  right  process  elements  be  identified  in  the  asset  base 
for  a  specific  project  as  a  function  of  product  requirements  and 
team  competence? 

Tailoring 

E-31 

Can  a  formal  approach,  based  on  a  sound  theoretical  basis, 
be  developed  to  address  the  tailoring  of  processes  for  specific 
implementations? 

E-32 

What  are  the  organizational  and  environmental  factors  that  affect 
tailoring  choices  (e.g.,  tailoring  for  small  enterprises  or  for  agile 
development)? 

E-33 

How  can  we  tailor  processes  with  predictable  effects  on 
efficiency? 

E-34 

How  can  processes  be  designed  for  easy  tailoring? 

E-35 

How  to  specify  processes  including  variability? 
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Interoperability 


E-36 

To  what  extent  is  it  possible  to  integrate  different  processes, 
in  particular  when  they  are  based  on  different  paradigms,  for 
example,  agile  processes  versus  more  waterfall-like  approaches? 

E-37 

How  can  we  harmonize  the  mental  model  of  sequential 
development  with  the  reality  of  continuous  iteration? 

E-38 

How  can  processes  be  packaged  together  with  a  quantitative/ 
qualitative  model  describing  their  behavior? 

E-39 

Are  there  mechanisms  for  understanding  and  improving 
interoperability  between  processes  (composability  analysis:  pre/ 
post  conditions,  inputs/outputs,  styles,  assumptions)? 

E-40 

How  can  domain-specific  development  processes  (product 
dependent)  and  software/systems  development  processes  be 
synchronized? 

Evolution 

E-41 

How  does  the  context  of  a  process  (e.g.,  organization  size,  culture, 
process  distribution)  influence  process  evolution? 

E-42 

How  can  we  evolve  process  lines  based  on  deployment  feedback? 

E-43 

How  can  processes  be  made  sufficiently  adaptive  to  provide 
effective  support  in  responding  to  domain-specific  change? 

4.5.3  Research  Node  E.3:  Providing  Process 
Engineering  Infrastructure 

The  objective  of  this  research  node  is  to  provide  infrastructure  to  sup¬ 
port  process  engineering.  This  infrastructure  includes  the  appropriate 
support  organization  and  integrated  support  for  all  the  activities  for 
specification,  integration,  tailoring,  and  learning. 

The  research  questions  are  divided  into  two  areas:  (1)  organization  and 
training  and  (2)  technology.  The  organization  questions  focus  on  defin¬ 
ing  organizational  hierarchies  that  reflect  the  advances  being  proposed. 
In  parallel  the  training  questions  focus  on  the  related  people  skills.  The 
technology  questions  focus  on  process  lines  and  other  technology  ap¬ 
proaches  to  support  organizing  processes  for  reuse. 
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Research  questions  associated  with  this  node  include 

Organization  &Training 


4 


E-44 

What  process  infrastructures  are  appropriate  to  support  the  new 
technologies  and  concurrent  engineering? 

E-45 

How  do  we  create  easy  to  use  "experience  bases"  that  allow 
knowledge  to  be  stored,  updated,  and  accessed  by  developers  at 
varying  levels  (to  facilitate  the  continuous  evolution  of  problem 
domains  and  technical  skills  required  as  businesses  move  into 
new  hybrid  domains,  for  example)? 

E-46 

How  do  the  team  competencies  affect  the  engineered 
development  processes? 

E-47 

How  do  we  educate  people  for  process  engineering  in  general  and 
the  use  and/or  development  of  process  lines  in  particular? 

E-48 

How  do  we  educate  people  in  the  need  for  and  the  use  of 
evidence? 

Technology 

E-49 

Which  activities  related  to  evidence  creation,  process  line 
engineering,  and  usage  can  or  should  be  automated? 

E-50 

What  automated  decision  support  is  useful  and  how  can 
automation  and  human  (educated)  creativity  be  balanced? 

E-51 

How  do  we  perform  process  model  configuration  management? 

E-52 

How  could  "process  patterns"  be  best  used? 

E-53 

What  is  the  role  of  process  simulation  in  providing  trust,  scaling, 
and  supporting  process  prediction,  selection,  tailoring,  and 
integration? 

E-54 

What  visualizations  can  support  process  management  (including 
different  views  for  different  stakeholders  and  different  business 
domains)? 

E-55 

What  level  of  statistical  analysis  is  feasible  and  reasonable  to 
apply  to  process  management? 

E-56 

What  is  appropriate  support  for  automated  metrics  collection? 

E-57 

How  can  an  inference  engine  for  effective  display  and  retrieval  of 
processes  be  constructed? 

E-58 

How  can  we  use  process  evidence  to  derive  theories  about 
process,  product,  and  resource  dependencies? 

E-59 

How  do  we  define  intellectual  property  for  a  process? 

E-60 

How  can  we  integrate  process  engineering  and  workflow 
management  tools? 
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Theme  P:  Managing 
Project  Processes 

This  theme  was  developed  by  Claes  Wohlin, 
Gonzalo  Cuevas,  IMidhi  Srivastava,  and  William 
Peterson. 


Research  Nodes  and  Questions  in  this  Theme 

•  ResearqfvNode  P.1: 

Operating  Under  Autonomous  Control 
(PI -PI  6) 

•  Research  Node  P.2: 

Managing  Through  Centralized  Cooperation 
(P17-P27) 


•  Research  Node  P.3: 

Agreeing  Under  Federated  Collaboration 
(P28-P32) 


5.1  Introduction  ofTheme 


A  large  number  of 
companies  practice 
distributed  or  global 
development  that 
could  most  likely  be 
done  more  effectively 
and  efficiently;  hence, 
applied  research  is 
needed. 

This  theme  is  primarily  concerned  with  the  evolution  and  impact  of  or¬ 
ganizational  control  structures  on  the  management  of  project  processes. 
This  theme  is  therefore  a  filter  for  the  other  core  research  themes  in  so 
far  as  it  provides  context  against  which  to  consider  the  research  ques¬ 
tions  in  each  of  the  other  themes.  In  addition,  this  theme  adds  research 
questions  related  to  managing  specific  project  processes. 

5.2  Theme  Rationale 

This  research  theme  is  concerned  with  the  formulation  and  evaluation  of 
processes  that  specifically  address  the  challenges  related  to  organizational 
control  structures  and  in  particular  their  effect  on  projects.  The  theme  is 
motivated  by  the  increased  use  of  distributed  development.  For  example, 
distributed  development  may  be  within  one  or  several  companies,  at  one 
or  multiple  locations,  with  one  decision  authority,  centralized  cooperation 
or  federated  collaboration,  and  with  shared  or  different  organizational 
goals.  The  distributed  development  may  include  different  time  zones 
and  different  cultures.  Such  globalization  will  continue.  However,  it  may 
take  different  shapes  depending  on  developments  in  the  world.  It  may 
include  more  traveling  or  less  travel,  but  the  requirements  to  be  able  to 
co-develop  from  separate  sites  toward  a  joint  development  base  will  keep 
increasing.  Development  “24/7/365”  (or  at  least  24/5+,  assuming  that  all 
sites  have  a  weekend)  may  become  more  and  more  common. 

This  research  theme  differs  from  the  others  in  the  sense  that  research  has 
not  been  sufficiently  translated  to  industry  practice,  and  consequently 
research  is  not,  or  at  least  not  yet,  driving  progress.  A  large  number  of 
companies  practice  distributed  or  global  development,  within  their  own 
organizations  or  using  outsourcing.  Companies  have  managed  to  “get  by” 
so  far,  but  distributed — and  especially  global — development  could  most 
likely  be  done  more  effectively  and  efficiently;  hence,  applied  research  is 
needed. 

Thus,  researchers  have  to  understand  research  in  related  fields  and  the 
current  industry  practice  and  start  to  improve  on  it,  taking  on  some  of  the 
key  challenges  with  global  software  development.  This  is  particularly 
true  of  free  and  open-source  software.  Research  in  related  fields  includes 
work  on  virtual  teams  and  cultural  issues.  This  research  has  not  yet  been 
fully  integrated  into  the  research  in  software  engineering,  which  may  be 
exemplified  by  a  new  conference  that  has  been  launched  in  the  area:  the 
International  Conference  on  Global  Software  Engineering  (ICGSE).10 

The  conference  presents  the  challenges  as  follows: 

10  More  information  can  be  found  at  http://www.inf.pucrs.br/icgse2006/. 
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Developing  software  across  borders  is  becoming  an  important 
competitive  advantage  in  today  s  software  industry.  However, 
the  increased  globalization  of  software  development  creates 
software  engineering  challenges  due  to  the  impact  of  time  zones, 
diversity  of  culture  and  communication,  or  distance.  This  requires 
novel  and  effective  techniques  and  behaviors  to  achieve  intended 
productivity  and  quality  targets. 

ICGSE  2006  is  the  first  International  Conference  that  aims  at 
bringing  together  researchers  and  industry  practitioners  to 
explore  both  the  state-of-the-art  and  the  state-of-the-practice  in 
software  engineering  for  global  software  development. 

Research  papers  are  published  in  the  area,  but  they  are  too  few  and 
several  of  them  are  pointing  to  the  challenges  and  problems  rather 
than  presenting  solutions.  A  search  in  the  ISI  database,  which  contains 
the  major  journals,  on  “global  software  development”  results  in  17 
papers  being  identified,  and  a  search  on  “outsourcing”  AND  “software 
development”  resulted  in  16  papers,  with  a  small  overlap  between  the 
two  searches.11  The  two  lists  are  provided  in  the  Further  Reading  and 
References  section  at  the  back  of  this  book.  Several  things  may  be  noted. 
The  oldest  paper  is  from  1996,  and  several  of  the  papers  belong  to  Lecture 
Notes  in  Computer  Science  or  the  special  issue  of  IEEE  Software  devoted 
to  global  software  development.  In  summary,  relatively  little  research  into 
global  software  development  has  so  far  been  published  in  journals. 

It  is  also  noteworthy  that  only  a  few  papers  are  published  a  year  in  the  in¬ 
ternational  journals  on  this  topic.  Although  the  information  is  not  included 
in  our  results  lists,  it  is  interesting  to  note  that  no  paper  from  the  first  list 
(from  a  search  on  “global  software  development”)  had  been  cited  more 
than  six  times  as  of  February  2006,  the  time  of  our  search.  This  in  itself  is 
an  indication  that  little  research  is  being  done  in  the  area. 

In  addition  to  the  articles  found  in  the  citation  database,  we  note  that  the 
journal  Software  Process  Improvement  and  Practice  devoted  a  special 
issue  to  global  software  development  in  2003. 12  The  journal  Information 
and  Software  Technology  planned  a  special  issue  on  global  software  de¬ 
velopment  in  2006.  However,  the  number  of  submissions  was  rather  low 
and  finally  only  two  papers  were  judged  to  meet  the  quality  standards. 

The  Association  for  Computing  Machinery  (ACM)  published  a  report  on 
global  development  and  outsourcing  in  March  2006  [Sapray  2006]. 

11  It  should  be  noted  that  the  ISI  database  is  normally  used  in  library  research, 
and  hence  it  is  a  good  source  for  sampling  research  articles. 

12  See,  for  example,  the  following  article:  Damiam,  D.  E.  "Global  Software  Development: 
Growing  Opportunities,  Ongoing  Challenges."  Software  Process  Improvement 

and  Practice  8, 4  (October/December  2003):  179-182. 
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Also,  IEEE  Software  published  a  special  issue  in  September/October 
2006  on  global  software  development  and  began  a  regular  column  on 
free-  and  open-source  software  in  2005. 

Global  software  development  provides  some  opportunities  but  also  a 
vast  number  of  challenges  (some  of  which  also  exist  for  single-site  de¬ 
velopment),  including 

•  Process  is  often  documented,  but  in  many  instances  another  process 
is  practiced.  How  should  we  be  able  to  coordinate  the  actual 
processes  used? 

•  How  do  we  manage  time  zones  and  different  cultures? 

•  Is  it  at  all  possible  to  work  with  a  common  development  base,  where 
software  is  checked  in  and  checked  out  by  different  development 
sites? 

•  How  do  we  manage  integration  of  software  developed  at  different 
sites? 

•  How  do  we  divide  software  development  among  sites? 

The  list  of  challenges  can  be  made  very  long.  The  objective  of  this  re¬ 
search  theme  is  to  try  to  structure  some  of  the  challenges  and  formulate 
research  needs  based  on  them. 

5.3  Characterizing  the  Current  State  of  the 
Practice 

Today  we  have  many  small  companies  operating  on  only  a  single  site. 
Some  of  these  companies  operate  collaboratively  with  others,  possibly 
to  enhance  capacity  or  to  avail  themselves  of  specialist  skill  sets.  These 
collaborations  may  involve  several  companies  in  one  country  or  com¬ 
panies  distributed  across  international  borders,  where  the  cost  base  or 
political  considerations  may  be  important  drivers.  Such  collaborations 
tend  to  be  managed  through  subcontractor  relationships. 

Many  larger  companies  are  split  across  multiple  sites,  again  either  in 
one  country  or  across  several  countries.  In  such  circumstances,  there 
may  often  be  a  greater  tendency  to  have  collaborative  working  ar¬ 
rangements  without  the  notion  of  a  subcontractor  management  struc¬ 
ture.  These  projects  may  be  considered  more  integrated  than  is  often 
achieved  through  a  contractual  relationship. 

The  level  of  process  integration  across  split  sites  tends  to  be  very  low 
(“geography  is  destiny”).  This  is  particularly  true  where  cross-site  man¬ 
agement  is  achieved  through  subcontractor  relationships.  Each  site  often 
deals  with  the  other  sites  as  “black  boxes,”  this  being  the  easiest  struc¬ 
ture  to  handle  given  the  current  lack  of  process  culture  that  pervades. 
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Where  higher  levels  of  integration  have  been  successful,  integration  is, 
nevertheless,  normally  organized  around  development  life-cycle  stages. 
For  example,  requirements  development  and  design  are  completed  on 
the  same  site.  Then  implementation  and  testing  are  accommodated  on 
different  sites.  True  blending  of  team  members  across  split  sites  is  very 
much  in  the  minority. 

Many  companies  run  projects  across  sites  in  different  countries. 
Research  is  being  conducted,  but  it  is  still  quite  limited  in  comparison  to 
the  wide  use  of  global  development  today,  based  on  the  searches  men¬ 
tioned  of  the  ISI  database.  Some  key  areas  of  research  may  be  identified 
and  exemplified  with  some  of  the  articles  found.  Research  areas  include 

•  coordination  of  global  development  (e.g.,  [Carmel  2001, 

Hersleb  2003,  Smite  2004,  Smite  2005]) 

•  virtual  teams  (e.g.,  [Jacobs  2005,  Sakthivel  2005]) 

•  agile  development  (e.g.,  [Reeves  2004,  Rottman  2006, 

Tiwana  2004]). 

•  a  number  of  articles  were  identified  when  searching  for 
“outsourcing”  (e.g.,  [Kishore  2004,  Dayanand  2001, 

Bhatnagar  1997]) 

Global  software  development  and  outsourcing  is  a  worldwide  trend, 
and  research  is  catching  up  with  it.  Evidence  to  support  this  conclusion 
include  the  inception  of  a  new  conference  devoted  to  global  develop¬ 
ment  (ICGSE),  ACM’s  report  on  the  topic,  and  the  increasing  frequency 
at  which  articles  on  the  topic  are  published  (much  more  frequently 
than  10  years  ago).  Having  said  this,  the  amount  of  research  into  global 
software  development  and  outsourcing  is  still  low  in  comparison  to  the 
extent  in  which  it  is  practiced  in  industry. 

5.4  Characterizing  the  Desired  State  of  the 
Practice 

In  this  state,  small  and  large  organizations  live  together  in  harmony 
and  have  culturally  infused  process  awareness.  Federations  of  small 
organizations  tend  to  grow  in  response  to  project  needs.  As  the  projects 
complete,  the  federations  reduce  in  size.  Even  large  organizations  tend 
to  manage  themselves  around  a  federated  system.  The  need  for  process 
in  this  environment  is  unquestioned.  In  fact,  process  compliance  is  a 
mandatory  requirement  for  entry  onto  the  “register  of  suppliers.” 

Continuous  process  improvement  is  a  regular  practice  at  organizations. 
Education,  training,  and  professional  awareness  support  the  above 
concepts.  Integrated  process  standards,  including  other  disciplines  and 
business  processes  for  different  domains  are  common.  Emerging  pro¬ 
cesses  for  systems  engineering  of  global-scale  complex  and  adaptive 
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systems  are  aligned  with  stakeholder  values.  The  use  of  process  simula¬ 
tions  based  on  various  parameters  to  reduce  risks  is  common.  Process  is 
viewed  as  an  enabler  for  cooperative  working. 

Not  only  do  organizations  evolve  rapidly  in  this  environment  to  meet 
business  needs,  but  so  too  do  their  individual  staff  complements.  The 
notion  of  a  job  for  life  has  almost  completely  gone.  Now  people  are 
brought  in  on  personal,  short-term  contracts,  according  to  project  work 
demands.  People  are  certified  against  processes  or  process  components. 
This  means  that  employers  can  ramp-up  teams  much  more  efficiently  in 
response  to  business  needs. 

5.5  Description  of  the  Research  Nodes 

The  focus  of  this  theme  is  from  a  “project”  perspective.  The  three  re¬ 
search  nodes  in  this  theme  are 

•  operating  under  autonomous  control  (P.  1 ) 

•  managing  through  centralized  cooperation  (P.2) 

•  agreeing  under  federated  collaboration  (P.3) 

The  nodes  represent  progression  from  a  research  perspective  and  an 
organizational  complexity  perspective.  Most  development  organizations 
would  move  through  the  three  nodes  as  a  progression.  Very  few  devel¬ 
opment  organizations  are  expected  not  to  pass  through  those  stages  of 
growth.  Moreover,  the  research  issues  become  more  complicated  as 
we  progress  through  the  nodes.  Thus,  it  is  reasonable  to  believe  that 
good  solutions  are  needed  for  one  node  before  moving  onto  the  follow¬ 
ing  nodes.  For  example,  if  someone  would  like  to  address  research  for 
Node  P.3,  the  research  questions  in  nodes  P.l  and  P.2  are  most  likely 
also  relevant.  (However,  in  presenting  the  nodes  in  this  document,  we 
do  not  repeat  research  questions  for  nodes  with  a  higher  number.) 

For  each  of  the  three  nodes,  research  has  to  address  how  open-source, 
multidisciplinary  competence  provisioning  and  experience  bases  are 
handled.  Furthermore,  the  processes  need  to  be  flexible  and  tailorable  to 
the  needs  and  the  size  of  the  development  (in  terms  of  partners  involved 
and  number  of  people  involved).  Moreover,  different  processes  at  dif¬ 
ferent  parts  of  a  company  or  across  multiple  companies  must  be  able  to 
interact.  The  interfaces  between  process  parts  become  crucial.  Results 
from  research  questions  could  help  us  in  structuring  project  work  appro¬ 
priately  for  projects  of  varying  sizes  and  distribution. 

For  the  nodes  below,  we  are  assuming  process-inclined  or  process-sav¬ 
vy  organizations.  We  do  not  know  how  process  maturity  of  an  organiza¬ 
tion  correlates  to  its  ability  to  move  and  progress  through  the  nodes. 
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5.5.1  Research  Node  P.1 :  Operating  Under 
Autonomous  Control 

The  process  objectives  of  this  node  are  to  focus  on  projects  with  au¬ 
tonomous  control  to  achieve  predictable  outcomes  in  terms  of  function¬ 
ality,  cost,  schedule,  and  quality  attributes  when  developing,  evolving, 
and  maintaining  software.  Autonomous  control  refers  to  situations 
where  there  is  one  decision  authority  and  everybody  works  according 
to  the  same  project  process  and  joint  organizational  goals.  However, 
it  does  not  exclude  distributed  development,  working  across  cultural 
boundaries,  or  having  different  companies  involved.  The  key  issue  is 
that  one  prime  has  the  decision  authority  and  that  anyone  involved  in 
the  project  uses  the  same  project  process  instantiation.  Some  examples 
of  this  situation  include  the  following: 

•  one  company  at  one  or  several  sites 

•  “body  shopping”  with  a  requirement  to  use  the  same  project  process 
instantiation 

•  several  companies  with  one  prime  and  everybody  uses  the  same 
project  process  instantiation 

In  this  node,  organizations  may  have  established  their  process  assets 
(process  repository)  and  their  improvement  program.  Also,  they  may 
have  established  the  competency  roles  and  profiles,  which  allow  them 
to  know  the  talents  of  the  people  and  thus  to  establish  the  most  effective 
structure  and  staffing  in  each  project.  In  addition,  the  process  models 
are  extended  to  cover  all  life-cycle  aspects:  products  and  services. 
Finally,  in  this  state  the  results  of  a  project  in  terms  of  cost,  functional¬ 
ity,  schedule,  and  quality  should  be  predictable. 

The  motivation  for  operating  projects  with  one  decision  authority 
and  using  the  same  project  process  instantiation  comes  from  several 
sources: 

•  reduction  of  overhead  for  communication  and  interaction  due  to  the 
use  of  the  same  project  process  instantiation  for  different  working 
parts  of  the  team 

•  the  ease  of  communications  and  enhanced  ability  to  share  process 
understanding,  though  in  this  environment  there  may  be  a  lower 
motivation  for  process  adoption 

•  other  motivations  for  this  organizational  structure  include  the  ability 
to  react  more  quickly  to  market  demands  and  also  the  ability  to  be 
close  to  particular  clients 
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Research  questions  associated  with  this  node  include13 


Process  Type  (including  scale) 


P-1 

How  do  we  select  the  best  possible  project  process  or  set  of 
processes  to  use?  The  latter  is  particularly  relevant  when  having 
different  processes.  Selection  of  the  project  process  is  a  key  issue 
to  be  able  to  achieve  project  goals.  Moreover,  it  must  be  possible 
to  adapt  the  process  to  different  types  of  applications,  project 
size,  and  so  forth. 

P-2 

How  do  we  scale  processes  to  our  needs?  How  does  a  subject 
matter  expert  (SME)  know  when  it  is  time  to  have  more  formal 
procedures  in  place?  SMEs  cannot  be  expected  to  have  well- 
defined  processes  in  place.  It  is  hence  important  to  be  able  to 
have  as  little  overhead  as  possible,  but  still  sufficient  to  enable 
the  delivery  of  high-quality  software.  As  a  small  company  grows, 
it  is  important  to  realize  that  the  processes  used  have  to  change. 

The  questions  are  when  to  change  them  and  how  to  change  them. 

It  is  easy  for  a  company  to  grow  out  of  its  process  support. 

Competency 

P-3 

What  are  the  needed  competencies  for  the  required  tasks  on  a 
specific  project?  How  is  the  relation  between  competence  profile 
and  development  process  handled?  It  is  a  challenge  to  have 
the  right  mix  of  competencies  in  a  given  project,  in  particular 
when  certain  type  of  competence  may  be  needed  in  too  many 
projects  at  the  same  time.  It  is  also  a  particular  problem  for  small 
companies  where  often  the  same  person  must  take  on  multiple 
roles.  This  issue  becomes  an  even  larger  challenge  when  looking 
into  successive  nodes  within  this  theme.  Furthermore,  how  do  you 
"process-bond"  teams  that  are  brought  together  from  outside  the 
company  for  specific  projects? 

P-4 

How  do  individual  competencies  sum  up  in  the  team?  The 
distribution  across  site  means  on  the  one  hand  that  competencies 
at  different  sites  become  available,  but  on  the  other  hand  it  may 
be  difficult  to  get  what  one  wants  from  a  project  perspective.  This 
may  be  because  the  site  manager  has  many  projects  and  would 
like  to  staff  them  according  to  an  optimization  for  the  site  and  not 
for  a  project. 

P-5 

How  do  we  make  optimal  use  of  available  competencies? 

Different  companies  mean  different  competencies.  How  do  we 
effectively  combine  competencies  that  are  available  in  different 
companies? 

13  We've  given  each  research  theme  a  letter  identifier  so  that  the  questions  associated 
with  each  theme  can  be  more  readily  identified.  We've  identified  this  theme  on  manag¬ 
ing  project  processes  with  the  letter  P.  Other  themes  on  the  relationships  between 
processes  and  product  qualities  (identified  with  Q),  process  engineering  (E),  and  process 
deployment  (D)  feature  the  same  treatment.  In  addition,  questions  associated  with  the 
effects  of  emerging  trends  are  identified  with  a  T  and  those  in  the  example  instantiation 
for  security  with  an  S. 
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P-6 

How  do  we  capture  and  share  experiences  across  sites?  Project 
work  means  building  experience,  which  is  difficult  to  share  in 
general,  but  becomes  even  more  complicated  when  the  team  is 
distributed  across  sites.  A  certain  understanding  or  experience 
may  come  at  a  site  internal  meeting,  but  how  is  this  experience 
communicated  to  others  not  at  the  meeting,  and  in  particular  if 
they  are  at  another  site? 

Distribution 

P-7 

How  do  we  manage  development  between  different  locations?  This 
includes  managing,  for  example,  responsibilities  and  risks  between 
locations  (including  risks  inherent  in  distributing  in  the  first  place). 
Development  across  locations  is  in  many  cases  a  necessity  for  large 
project  and  large  companies.  This  means  that  the  actual  distribution 
of  the  development  must  be  managed.  What  management 
procedures  are  needed  to  manage  a  distributed  project?  How  is  time 
reporting  done?  To  what  extent  do  things  have  to  be  done  in  the 
same  way  at  the  different  locations? 

P-8 

How  do  we  divide  the  work  effectively  and  efficiently  between 
locations?  The  work  should  not  only  be  divided  between  the 
locations,  it  should  be  divided  in  an  effective  and  efficient  way.  This 
includes  taking  the  current  architecture  into  account 

P-9 

How  do  we  distribute  quality  requirements?  The  distribution  of 
work  often  means  that  functionality  is  distributed,  but  how  do  we 
handle  non-functional  requirements?  The  customer  or  market  has 
expectations  on  certain  quality  aspects  for  the  whole  system,  but 
when  distributing  the  work  it  may  mean  that  quality  requirements 
are  forgotten  or  it  is  very  difficult  to  break  them  down  to  parts  of  the 
software.  A  typical  example  is  how  to  handle  performance. 

P-10  How  do  we  handle  different  time  zones?  Times  zones  are  obstacles 

for  communication,  but  also  an  opportunity  for  having  development 

24  hours  a  day.  This  includes  challenges  related  to 

•  How  do  we  handle  time  zones  and  how  do  we  effectively 
communicate  the  results  from  one  site  to  another?  How  do  we 
make  use  of  differences  in  time  zones?  This  question  is  about 
the  opportunities.  How  can  different  time  zones  help  us  develop 
software  more  efficiently?  Is  it  possible  to  work  24  hours  a 

day  with  development  and  pass  assignments  around?  Can 
technologies  such  as  Instant  Messaging  and  Voice  over  IP  make 
the  world  one  global  workplace? 

•  What  is  the  productivity  drag,  if  any  due  to  time  zone 
differences?  Is  the  productivity  different  in  different  sites?  Why? 
How  can  we  leverage  form  the  best?  Do  people  in  different 
cultures  make  different  types  of  mistakes? 

•  How  do  we  work  toward  a  joint  base  (configuration 
management)?  Many  configuration  management  tools  exist,  but 
are  they  able  to  cope  with  potential  issues  such  that  developers 
in  different  sites  wanting  to  work  on  the  same  component  at  the 
same  time? 

•  How  do  we  manage  different  cultures  and  time  zones  between 
different  companies?  Different  companies  imply  different 
processes.  Distributed  development  around  the  globe  also 
implies  different  cultures.  How  do  we  handle  the  mix  of  cultures 
and  mix  of  processes? 
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Cultural 


P-11  How  do  we  handle  cultural  differences?  Cultural  differences  are 
a  fact.  The  challenge  is  first  to  be  aware  of  them  and  then  to  use 
them  to  our  advantage.  Can  formal  training  deal  with  the  above 
challenge?  How  do  we  educate  people  to  work  in  global  software 
development?  What  additional  knowledge  is  needed  to  ensure 
that  developers  can  work  in  a  global  environment?  It  does  not  only 
require  knowledge  about  software  development,  but  also  a  good 
understanding  of  the  challenges  with  global  development  and  with 
other  cultures. 


P-12  How  do  we  ensure  the  same  process  interpretation  in  different 
cultures?  Can  Enterprise  Process  Frameworks  address  the  issues 
of  multiple  process  styles?  How  heavy  or  light  should  an  enterprise 
framework  be  to  meet  business  objectives?  Even  when  having 
the  same  process  across  sites,  there  is  no  guarantee  that  it  is 
interpreted  in  the  same  way.  The  likelihood  is  higher  if  being 
within  one  country  and  with  people  with  a  similar  educational 
background,  but  it  becomes  much  more  a  challenge  when  having 
people  form  different  cultures. 


P-13  How  do  we  make  use  of  experiences  in  one  culture  in  another 
place?  It  is  well-known  that  it  is  difficult  to  share  experiences 
effectively.  However,  it  becomes  even  more  complicated  to  share 
and  use  experiences  in  one  culture  with  another  culture.  How  can 
this  be  done?  It  is  probably  insufficient  to  publish  experiences;  they 
have  to  be  transferred,  but  what  is  the  best,  most  effective  way? 


P-14  How  can  the  end-user  perspective  be  captured  in  global  software 
development?  The  end-user  comes  further  away  from  the 
development  when  the  development  goes  global.  Thus,  it  may 
become  even  more  difficult  to  include  the  end-user  perspective 
into  the  development. 


Standards  and  Regulations 

P-15  How  do  we  make  processes  that  are  compliant  with  accepted 
standards?  Different  companies  in  different  countries  may  have 
different  standards  or  they  may  even  be  required  to  follow 
different  standards  due  to  legislation.  How  can  global  software 
development  be  conducted  with  different  standards? 


P-16  Are  global  standards  an  imperative  for  this  model  to  succeed?  Is 
it  possible  to  succeed  with  global  development  without  having 
common  standards?  Can  global  process  interface  standards  be 
developed? 
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5.5.2  Research  Node  P.2:  Managing  Through 
Centralized  Cooperation 

The  process  objectives  of  this  node  are  to  focus  on  projects  with  one 
main  decision  authority,  but  using  different  project  process  instantia¬ 
tions.  These  projects  are  most  likely  decentralized  and  made  up  of  a 
possible  mix  of  mature  and  immature  organizations,  but  having  joint 
organizational  goals.  Some  examples  of  this  situation  include:  one  com¬ 
pany  in  multiple  sites,  divisions  and  so  forth;  and  subcontracting  with 
one  prime. 

Scenarios  within  this  node  could  be  large  projects  in  one  company  with 
multiple  development  locations  or  multiple  companies.  The  reasons  for 
having  development  in  multiple  locations  are  many:  access  to  labor, 
cost-reduction,  presence  in  a  specific  country  or  region,  tax  incentives 
to  establish  work  based  in  economically  disadvantaged  parts  of  a  coun¬ 
try,  or  where  there  are  pockets  of  expertise  (e.g.,  in  regions  surrounding 
a  University  with  a  good  reputation  in  certain  fields).  Even  the  need  for 
a  sufficient  number  of  people  can  cause  companies  to  establish  them¬ 
selves  at  multiple  locations  or  to  cause  subcontracting  to  other  compa¬ 
nies.  Other  reasons  include  having  locations  near  major  clients  or  near 
important  suppliers. 

The  objective  is  to  support  companies  in  being  able  to  divide  projects 
into  suitable  parts  that  can  be  developed  on  different  locations  and  then, 
as  easily  as  possible,  be  integrated  into  a  system.  This  is  a  challenge, 
and  it  will  make  more  demands  of  process  than  perhaps  a  single  loca¬ 
tion  operation,  however  it  has  a  better  chance  of  succeeding  where  the 
processes  and  culture  support  a  joint  understanding. 

Research  questions  associated  with  this  node  include 


Different  Processes 


P-17 

How  do  we  handle  the  use  of  different  project  processes?  Different 
locations  may  have  different  project  processes  or  at  least  different 
flavors  of  the  same  process.  This  poses  some  challenges  that  are 
reflected  in  the  remaining  questions  in  this  group. 

P-18 

Is  the  output  from  one  process  producing  the  input  needed  in 
another  process,  or  are  the  differences  between  processes  at 
different  locations  a  problem? 

P-19 

How  do  we  handle  interfaces  between  processes?  The  interface 
between  different  processes  in  distributed  development  must  be 
well  specified  and  the  output  from  one  process  should  be  the  input 
to  another  process. 
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P-20 

How  are  the  abilities  to  deal  with  multiples  process  style 
managed?  For  example,  is  it  possible  to  combine  a  waterfall 
approach  in  one  location  with  a  much  more  agile  approach  in 
another  location?  What  requirements  have  to  be  put  on  the 
development  to  ensure  that  we  obtain  one  integrated  system  at 
the  end  of  the  day? 

P-21 

How  can  we  handle  companies  with  different  process  capability? 

How  do  customers,  suppliers,  and  vendors  with  different  process 
capabilities  work  effectively?  Is  the  process  capability  determined 
by  the  lowest  common  denominator? 

P-22 

How  do  we  integrate  open  source  code  in  company-specific 
products?  Open  source  is  often  used  as  "good  example."  How  can 
software  from  open  source  be  used  in  commercial  context?  How 
can  we  learn  from  how  open  source  development  is  conducted? 

Team  issues  and  Competence 

P-23 

How  do  we  manage  virtual  teams?  How  are  virtual  teams  formed? 
How  are  competencies  found  and  combined?  A  virtual  team  is  here 
defined  as  a  team  of  individuals  working  together  in  a  specific  role, 
for  a  specified  time  and  with  a  specific  goal,  however  without  any 
supporting  organizational  structures. 

Central  Authority 

P-24 

How  do  we  identify  suitable  partners/subcontractors?  How  are 
the  relationships  between  different  roles  managed?  The  problem 
here  is  about  identifying  the  most  suitable  partners  to  meet  our 
goals.  This  includes  identifying  suitable  collaborative  partners  and 
subcontractors.  What  is  the  role  of  SMEs  in  these  relationships? 

P-25 

How  do  we  put  requirements  on  subcontractors?  What  is  a  minimal 
set  of  requirements  when  subcontracting  internationally?  This 
includes  both  functional  and  non-functional  requirements. 

P-26 

How  do  we  accept  components  delivered  from  a  subcontractor? 
Delivered  software  must  be  accepted.  How  do  we  perform 
acceptance  testing?  This  may  be  particularly  difficult  when  only 
part  of  a  system  is  delivered.  How  do  we  certify  a  specific  quality 
level  for  a  component  or  subsystem  of  the  final  system?  Particularly 
where  the  overall  product  qualities  may  not  be  directly  traceable  to 
that  'component'? 
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P-27  Do  locations  need  a  primary  and  secondary  role  based  on 

competencies  within  a  company  too?  Should  one  location  act  as 
the  primary  location  and  other  work  as  subcontractors?  The  actual 
roles  and  responsibilities  must  be  clearly  defined.  Should  sites  be 
structured  for  growth  as  "Centers  of  Excellence"? 


5.5.3  Research  Node  P.3:  Agreeing  Under  Federated 
Collaboration 

The  process  objectives  of  this  node  are  to  focus  on  projects  with  shared 
decision  authority  and  project  goals  but  with  different  organizational 
goals  and  processes.  These  projects  are  most  likely  decentralized 
made  up  of  a  possible  mix  of  mature  and  immature  organizations. 
Furthermore,  the  objective  is  to  create  effective  and  efficient  systems 
development  independent  of  times  zones,  cultures,  and  other  differences 
(for  example,  development  processes).  To  truly  harness  the  possibilities 
from  global  development,  the  best  companies  around  the  globe  must 
be  involved.  The  objective  must  be  to  create  strategic  alliances  with  the 
best  players  to  be  competitive.  Some  examples  of  this  situation  are  joint 
ventures  and  independent  development  with  one  integrator. 

One  scenario  of  this  node  is  to  make  use  of  different  expertise  at  dif¬ 
ferent  companies.  For  example,  there  could  be  a  split  between  domain 
knowledge  or  expertise  (provided  by  one  partner)  and  systems  and 
software  engineering  skills  (provided  by  another  partner).  The  objec¬ 
tive  could  be  to  create  virtual  value-added  networks  by  combining  the 
expertise  of  different  companies.  In  other  words,  several  companies 
jointly  develop  a  product,  where  there  is  an  agreement  about  how  to 
divide  income  from  the  sales. 

Research  questions  associated  with  this  node  include 

P-28  How  should  relationships  between  partners  be  formulated? 

Joint  development  of  a  system  means  having  some  explicitly 
agreed-upon  relationship.  The  relationships  include  other  sites, 
outsourcing,  partner  development,  and  networks  of  partners. 

The  networks  may  be  formed  for  a  specific  project  or  may  be 
more  long-term  and  go  beyond  the  scope  of  any  one  project.  It  is 
important  to  decide  what  the  core  assets  are  and  what  is  suitable 
to  let  others  handle. 

P-29  How  can  we  handle  interfaces  among  several  small  companies? 

To  compete  with  larger  companies  we  may  see  networks  of  small 
companies  working  together.  They  are  often  less  mature  and 
they  may  not  have  well-documented  processes.  How  do  small 
companies  together  become  mature  and  able  to  handle  larger 
projects? 
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P-30  How  should  a  process  for  collaborative  development  be 

formulated?  The  development  at  different  companies  requires 
some  process  for  the  actual  collaboration.  How  should  it  be 
handled? 


P-31  How  do  we  handle  change?  Requirements  change  during 

development.  This  becomes  much  more  cumbersome  when  a 
change  may  affect  not  only  one  company,  but  several.  What 
routines  need  to  be  in  place  to  handle  change  across  companies  in 
distributed  development? 


P-32  How  do  we  create  value-based  networks  of  partners  to  work  as 
peers  in  projects?  How  do  we  establish  shared  goals?  Different 
companies  may  have  different  goals,  but  to  be  successful  shared 
goals  are  needed.  Is  it  possible  to  agree  on  common  goals? 
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Theme  D: 

Process  Deployment 


This  theme  was  originally  developed  by  Terry  Rout 
Hanna  Oktaba,  Hans  Juergen  Kugler,  and  John 
Goodenough.  Stan  Rifkin,  Suzanne  Garcia,  and 
Eileen  Forrester  made  additional  contributions. 


Research  Nodes  and  Questions  in  this  Theme 

•  Research  Node  D.1 : 

Establishing  the  Infrastructure  (D1-C 

•  Research  Node  D.2: 

Motivating  Process  Use  (D7-D19) 

•  Research  Node  D.3: 

Effective  Adoption  and  Deployment 

—D.3. 1 :  Formulating  Process  and  Deployment 
Requirements 
(D20-D30) 

-D.3. 2:  Supporting  Effective  Adoption 
(D31-D35) 

-D.3. 3:  Assessing  Effectiveness 
(D36-D45) 

•  Research  Node  D.4: 

Managing  Ongoing  Process  Deployment 
(D46-D60) 


)6) 
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6.1  Introduction  ofTheme 


Process  deployment,  at  its  heart,  is  about  getting  processes  into  actual 
practice.  There  is  no  shortage  of  good  software  engineering  ideas,  but 
few  are  in  practice. 

The  process  deployment  theme  has  many  interrelationships  with  both 
the  product  and  process  themes  already  described.  The  main  inter¬ 
relationships  are  depicted  in  the  core  framework  structure  (Figure  3) 
presented  at  the  beginning  of  this  document.  Process  deployment  draws 
heavily  on  process  engineering.  Process  engineering  provides  the  pro¬ 
cesses  and  process  performance  models  that  process  deployment  is 
responsible  for  getting  into  practice.  Process  deployment  can  capture 
empirical  evidence  and  patterns  of  experience  from  the  deployment  and 
use  of  processes.  This  evidence  and  experience  base  is  fed  back  into 
process  engineering  and  product  quality  to  maintain  and  refine  process 
engineering.  In  fact,  it  is  an  open  research  question  about  the  cause  and 
effect  relationship  between  process  engineering  and  deployment:  which 
informs  the  other? 

The  scope  of  process  deployment  is  worth  exploring.  In  essence,  we  are 
suggesting  that  it  is  “deploy  x,”  where  x  is  any  software  engineering  re¬ 
lated  process  or  product.  The  reason  for  the  wide  scope  is  another  open 
research  question:  how  is  software  engineering  deployment  informed 
by  the  extensive  information  available  about  deploying  processes  and 
products  outside  of  software  engineering? 

6.2  Theme  Rationale 

This  theme  is  centered  on  people  at  all  levels  of  analysis:  individuals, 
teams,  groups,  organizations,  countries,  and  cultures.  We  are  focusing 
on  the  organizational  level  of  analysis  here.  Specifically,  we  are  con¬ 
cerned  with  how  collectives  become  aware  of,  make  sense  of,  and  use 
processes  in  their  daily  work;  how  to  ensure  processes  get  followed, 
and  how  to  measure  relevant  aspects  of  the  usage.  It  incorporates  issues 
dealing  with  education  and  training  to  develop  collective  competen¬ 
cies;  motivation  of  process  performers  to  use  and  faithfully  adhere  to 
the  agreed  process;  actual  usage  of  a  developed  or  standardized  process 
into  an  organization’s  projects  and  operations;  setting  and  measurement 
of  the  achievement  of  process  adoption  goals,  and  appraisal  of  the  de¬ 
ployed  processes  to  determine  their  fidelity  and  capability,  as  well  as  to 
evaluate  whether  the  usage  achieved  the  goals  of  adding  value  promised 
by  the  process  or  product. 
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Readers  need  to  recognize  the  critical  difference  between  process  im¬ 
plementation  and  process  deployment.  The  concept  of  deployment  goes 
beyond  the  single  instantiation  of  an  implemented  process,  to  address 
the  effective  deployment  of  a  process  specification  to  achieve  multiple 
implementations  of  the  process  across  an  organization,  each  tailored  to 
suit  its  specific  organizational  context.  The  successful  implementation 
of  a  process  instance  establishes  its  basic  functionality;  its  effective 
deployment  establishes  its  true  value  to  the  implementing  organization. 


Process  deployment 
extends  beyond  the 
single  instantiation 
of  a  process 
specification. 


Note  also  that  the  viewpoint  taken  here  is  primarily  that  of  the  transi¬ 
tion  agent,  the  role  which  is  responsible  for  understanding  the  adoption 
context,  for  creating  or  obtaining  relevant  transition  mechanisms  to 
support  that  context  (including  establishing  the  infrastructure  in  which 
the  transition  agent  “lives”)  and  for  establishing  and  measuring  progress 
toward  adoption  goals. 

6.3  Characterizing  the  Current  State  of  the 
Practice 

Almost  nothing  is  known  empirically  about  the  most  effective  ways 
to  deploy  software  engineering  processes.  Even  the  most  rudimentary 
questions  cannot  be  answered:  how  much  change  can  we  introduce  and 
how  fast?  Nor  do  we  know  how  to  estimate  the  duration  and  resources 
required  to  convert  an  idea  to  use.  We  know  almost  nothing  about  how 
to  set  transition  goals  and  we  are  at  the  beginning  of  our  knowledge 
about  how  to  measure  adoption,  that  is,  the  taking  up  of  new  ideas  and 
adapting  them  into  practice. 


We  do  not  know  how  to  take  the  knowledge  of  adoption  in  one  organi¬ 
zation  and  adapt  it  to  others.  We  do  not  know  what  of  the  technology  to 
be  introduced  has  to  be  altered  to  make  it  fit  better  with  the  organization 
into  which  we  desire  to  insert  it;  we  do  seem  to  know  more  about  the 
changes  we  need  to  make  to  the  organization  in  light  of  the  demands  of 
the  new  technology. 


As  a  practical  matter,  we  do  not  agree  on  how  software  engineering  or¬ 
ganizations  and  their  surrounding  internal  and  external  structures  work 
and  interact  (in  a  way  that  is  relevant  to  adoption). 

We  do  not  agree  on  what  motivates  organizations  to  change,  so  we  see 
work  on  quantifying  benefits,  sharing  details  (stories)  of  implementa¬ 
tions,  and  models  or  roadmaps  of  specific  implementations,  and  we  do 
not  know  if  they  are  necessary  or  even  beneficial. 
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Current  methods  to  measure  process  adoption,  such  as  standardized 
assessments,  appraisals,  and  evaluations,  are  disruptive  and  highly 
invasive.  They  are  heavily  reliant  on  staff  and  are  consequently  costly 
both  in  terms  of  external  consultants  and,  more  importantly,  in  terms  of 
internal  disruption  to  day-to-day  operations.  The  still-heavy  reliance  on 
human  judgment  for  most  appraisal  types  also  makes  conducting  con¬ 
sistent  and  reliable  appraisals  challenging. 

For  those  organizations  that  do  make  the  leap  of  faith,  or  that  have  been 
motivated  by  business  drivers  such  as  perceived  marketing  advantage  or 
the  ability  to  increase  their  customers’  confidence  levels  in  their  perfor¬ 
mance,  process  deployment  has  often  been  an  ad-hoc  procedure,  offer¬ 
ing  little  opportunity  for  insight  into  the  reasons  why  the  deployment 
was  successful,  or  conversely,  a  poor  understanding  of  why  the  initia¬ 
tive  failed.  Although  techniques  and  models  have  been  promulgated  to 
support  understanding  of  organizational,  group,  and  individual  behavior 
during  a  process  adoption,  research  in  these  approaches  is  sporadic  and 
significant  gaps  in  knowledge  of  effective  approaches  is  seen  in  differ¬ 
ent  system  domains. 

We  do  not  know  the  best  infrastructure  for  process  improvement. 
Software  engineering  process  groups  (SEPGs)  have  been  part  of  the 
dictum  [Fowler  1990],  but  their  effectiveness  has  not  been  objectively 
measured.  Further,  alternatives  to  SEPGs  been  not  been  systematically 
proposed  and  evaluated.  In  particular,  we  do  not  have  an  adequate 
contingency  view:  Under  what  conditions  do  the  various  infrastructure 
alternatives  perform  best  and  why? 

6.4  Characterizing  the  Desired  State  of  the 
Practice 

We  would  understand  how  organizations  work  and  what  motivates  them 
to  change,  to  improve.  We  would  be  able  to  assess  how  much  change  is 
reasonable  and  how  quickly  it  could  be  introduced.  And  we  would  be 
able  to  estimate  the  resources  and  duration  to  accomplish  the  targeted 
level  of  adoption. 

We  would  know  what  we  would  have  to  tailor,  both  with  respect  to  new 
the  process  technology  and  the  organization.  Our  tailoring  guidelines 
would  be  sensitive  to  a  range  of  contexts,  including,  but  not  limited  to, 
competitive  landscape  (e.g.,  time-to-market  pressure),  national  culture, 
organizational  culture,  professional  culture,  external  and  internal  driv¬ 
ers,  organizational  strategy,  and  rewards  and  reinforcement. 
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Accordingly,  processes  will  be  deployed  effectively  and  flexibly,  em¬ 
ploying  empirically  confirmed  tool  support  to  implement  flexible  and 
responsive  tailoring  of  process  models.  The  performance  of  processes 
will  be  enhanced  by  capturing  and  analyzing  performance  data  and  by 
improving  processes  based  on  that  data  and  analysis.  We  will  have  the 
capability  to  define  a  common  basis  for  capturing,  storing,  compar¬ 
ing,  and  sharing  experiences  in  implementing  processes  that  links  to 
common  approaches  to  process,  organizational,  and  implementation 
modeling.  We  will  have  frameworks,  grounded  in  appropriate  theory, 
for  knowledge  acquisition  from  process  implementation  experiences 
and  a  theoretical  basis  for  relating  results  of  process  implementation  to 
the  formal  specifications  and  organizational  context  into  which  the  pro¬ 
cesses  are  being  deployed. 

And  we  will  know  which  kind(s)  of  deployment  infrastructure  is  ap¬ 
propriate  and  how  much  investment  would  be  necessary  to  achieve  the 
desired  level  of  adoption. 

6.5  Description  of  the  Research  Nodes 

There  are  six  research  nodes  in  this  theme: 

•  establishing  the  infrastructure  to  be  the  focal  point  for 
deployment  (D.l) 

•  motivating  process  use  (D.2) 

•  formulating  process  and  deployment  requirements  (D.3 . 1 ) 

•  supporting  effective  adoption  (D.3. 2) 

•  assessing  effectiveness  (D.3. 3) 

•  managing  ongoing  process  deployment  (D.4) 

The  research  in  this  theme  supports  several  activities  needed  when  de¬ 
ploying  processes  and  measuring  their  effectiveness  in  an  organization: 

•  Establish  the  desire,  need,  and  fit  for  the  new  process,  for  the  change. 
Then  establish  sponsorship  for  deploying  those  processes,  aligning 
processes  to  business  goals,  and  investing  in  process  infrastructure 
and  process  improvement  (relates  to  research  nodes  “establishing  the 
infrastructure”  and  “motivating  process  use”). 

•  Establish  the  process  needs  within  the  organization  context.  These 
needs  are  satisfied  by  the  capabilities  of  the  process  engineering 
theme  in  the  framework  (relates  to  research  node  “formulating 
process  requirements”). 

•  Adopt  the  engineered  processes  based  upon  the  adoption 
infrastructure  (relates  to  research  node  “supporting  effective 
adoption”). 
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•  Evaluate  the  processes  currently  implemented  within  the 
organization  and  apply  the  results  to  characterizing  the  success  of 
adoption,  as  well  as  diagnosis  and  improvement  of  processes  to 
better  support  the  business  goals  (relates  to  research  node  “assess 
effectiveness”). 

•  Manage  the  ongoing  deployment  of  the  processes  within  the 
organization  ensuring  that  performance  and  capabilities  satisfy  the 
organization's  process  needs  (relates  to  research  node  “managing 
ongoing  process  deployment”). 

6.5.1  Research  Node  D.1 :  Establishing  the 
Infrastructure 

Transition  agents  are  responsible  for  establishing  the  mechanisms 
to  search  for  potential  process  improvements  and  process  improve¬ 
ment  models,  translate  those  potentials  into  solutions  for  their  own 
organization’s  problems,  find  sponsors  to  tailor  the  solution  and  tailor 
organization  to  the  new  process,  pilot  test  the  new  process,  collect  per¬ 
formance  adoption  information,  and  improve  the  deployment  process 
going  forward.  What  is  the  best  infrastructure  with  which  to  accomplish 
those  goals? 

SEPGs  have  been  proposed,  and  research  needs  to  be  conducted  to  iden¬ 
tify  and  validate  the  best  forms  of  SEPGs  or  alternative  infrastructures. 


Research  questions  associated  with  this  node  include14 


D-1 

What  infrastructures  have  been  tried  and  what  are  their  relative 
merits? 

D-2 

What  is  the  best  advice,  then,  on  which  infrastructure  fits  which 
(contingent)  context? 

D-3 

Where  should  the  infrastructure  sit  inside  the  served  organization? 
What  is  the  best  governance  model? 

D-4 

How  much  responsibility  for  the  adoption  cycle  is  it  appropriate 
and  effective  for  it  to  take  as  its  charter? 

D-5 

What  is  the  profile  of  the  best-suited  people  to  staff  the 
infrastructure  roles? 

14  We've  given  each  research  theme  a  letter  identifier  so  that  the  questions  associated 
with  each  theme  can  be  more  readily  identified.  We've  identified  this  theme  on  and  pro¬ 
cess  deployment  with  the  letter  D.  Other  themes  on  the  relationships  between  processes 
and  product  qualities  (identified  with  Q),  process  engineering  (E),  and  managing  project 
processes  (P)  feature  the  same  treatment.  In  addition,  questions  associated  with  the 
effects  of  emerging  trends  are  identified  with  a  T  and  those  in  the  example  instantiation 
for  security  with  an  S. 
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D-6 

What  is  the  quantitative  relationship  between  investment  in 
infrastructure  and  the  achievement  of  adoption  goals? 

6.5.2 

Research  Node  D.2:  Motivating  Process  Use 

We  have  swept  into  this  node  most  of  the  fundamental  questions  related 
to  adoption  because  absent  their  answers  we  cannot  hope  to  address 
the  questions  of  how  to  motivate  a  change  in  what  we  do  every  day  in 
a  software  engineering  organization.  We  are  particularly  cognizant  that 
we  have  to  keep  the  unit  of  analysis  (organizations)  in  mind  as  we  pro¬ 
pound  our  questions,  as  there  is  already  an  abundance  of  work  at,  say, 
the  individual  (psychological)  level.  It  is  well  settled  that  the  psycho¬ 
logical  level  does  not  scale  to  the  collective,  so  we  have  to  be  careful 
to  formulate  appropriate  questions  and  be  attentive  to  the  sources  and 
methods  we  use  to  answer  them. 

Research  questions  associated  with  this  node  include 

D-7 

How  do  organizations  work  (in  the  ways  relevant  to  adoption)? 

D-8 

How  do  organizations  change  and  improve? 

D-9 

Assuming  there  is  more  than  one  answer  to  the  first  question  and 
to  the  second,  how  do  we  map  the  ways  in  which  organizations 
work  to  the  ways  in  which  they  change? 

D-10 

How  do  we  estimate  the  resources  and  duration  needed  to  make 
planned  change? 

D-11 

Is  there  a  way  to  assess  the  degree  to  which  an  organization  is 
ready  to  make  a  change  to  an  extent  we  specify,  both  in  depth  and 
rate  (how  much,  how  soon)? 

D-12 

What  is  the  best  way  to  characterize  the  environment  of  the 
organization  that  is  relevant  to  understanding  and  tailoring  the 
deployment? 

D-13 

Is  there  a  difference  in  deploying  process  vs.  product? 

D-14 

What  is  the  best  way  to  align  and  increase  the  motivation 
and  ability  of  personnel  to  adopt  new  processes?  In  fact,  what 
motivates  change  for  each  type  of  change  target?  For  example,  are 
business  cases  relevant? 

D-15 

What  is  the  best  way  to  plan  the  tailoring  of  the  new  processes  in 
light  of  the  answers  to  all  of  the  questions  above?  To  tailoring  the 
organization? 
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D-16 

What  organization-based  confidence  or  acceptance  criteria  ensure 
that  when  defined  processes  are  adopted,  the  desired  results 
will  be  produced?  (These  criteria  are  used  to  decide  when  the 
engineered  processes  produced  by  the  Process  Engineering  layer 
are  worth  deploying  in  the  organization.) 

D-17 

How  do  we  determine  and  coherently  express  understanding 
of  the  cause  and  effect  relationship  among  processes/process 
changes,  deployment,  and  product  outcomes? 

D-18 

What  explains  the  difference  in  the  rate  and  success  of 
deployment  (i.e.,  small  vs.  large,  government  versus  commercial, 
business  sector  differences,  etc.)? 

D-19 

What  are  they  key  strategies  and  tactics  to  motivate  process  use 
in  settings  where  process  adoption  has  failed  in  the  past? 

6.5.3 

Research  Nodes  D.3:  Effective  Adoption  and 
Deployment 

The  objective  of  this  group  of  three  research  nodes  is  to  provide  struc¬ 
ture  and  insight  into  effective  process  deployment  activities.  Process 
deployment  includes  identification  of  process  needs  for  a  given  context; 
specification,  engineering,  and  refinement  of  processes  through  model¬ 
ing  and  simulation  (as  expressed  in  the  Process  Engineering  section); 
deployment  of  processes  through  a  variety  of  adoption  support  mecha¬ 
nisms  and  structures;  and  measurement  of  both  the  effectiveness  of  the 
process  in  meeting  the  organization’s  business  goals,  as  well  as  mea¬ 
surement  of  the  progress  and  success  of  the  adoption  itself. 

Deployment  includes  two  facets.  The  act  of  deploying  selected  techni¬ 
cal  and  management  processes  to  an  organization  or  project  is  the  first, 
and  deciding  which  deployment  processes  should  be  used  to  facilitate 
the  adoption  of  the  selected  processes  is  the  second.  These  facets  break 
into  three  activities: 

•  Formulate 

This  activity  determines  the  technical  and  management  process  and 
transition  needs,  as  well  as  the  competencies  of  the  human  resources, 
since  these  competencies  affect  the  ability  of  the  organization  to 
successfully  adopt  selected  processes. 

•  Adopt 

This  activity  introduces  the  selected  processes  into  the  organizational 
context. 

•  Assess 

This  activity  provides  evidence  showing  to  what  extent  the  selected 
processes  are  being  adopted  (i.e.,  used  and  followed  accurately),  and 
if  followed,  whether  the  process  is  achieving  its  expected  results. 
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The  “assess”  activity  identifies  the  effectiveness  of  the  adoption  (e.g., 
the  breadth  and  depth  of  deployment)  as  well  as  the  effectiveness 
of  the  deployed  processes  in  achieving  the  expected  organizational 
benefits. 

The  research  nodes  below  elaborate  research  needed  in  each  of  these 
areas  to  be  able  to  achieve  the  future  state  described  in  section  6.4. 

6.5.4  Research  Node  D.3.1:  Formulating  Process  and 
Deployment  Requirements 

The  process  research  objective  associated  with  this  node  is  two-fold:  (1) 
improved  methods  are  available  to  provide  a  high-level  description  of  an 
organization’s  process  needs  and  the  relationships  among  these  needs; 
and  (2)  improved  methods  are  available  to  efficiently  determine  require¬ 
ments  for  successful  deployment. 

The  business  needs  and  organizational  context  (e.g.,  barriers  and  en¬ 
ablers  for  deployment)  determine  the  requirements  that  deployed  pro¬ 
cesses  should  satisfy.  This  node  in  the  deployment  cluster  deals  with  re¬ 
search  issues  associated  with  determining  the  requirements  for  processes 
to  be  deployed,  as  well  as  understanding  and  expressing  the  requirements 
for  successful  deployment  of  the  selected  processes.  The  process  require¬ 
ments  are  to  be  satisfied  by  the  technology  called  for  by  process  engi¬ 
neering.  Process  engineering  provides  the  technology  for  selecting  and 
tailoring  processes  that  satisfy  an  organization’s  process  needs. 

In  addition  to  deciding  what  processes  need  to  be  deployed,  an  organiza¬ 
tion  needs  to  decide  which  deployment  processes  should  be  used,  again 
based  on  requirements.  This  node  also  addresses  research  questions 
related  to  improving  understanding  of  the  requirements  for  different  de¬ 
ployment  approaches  for  different  organizational  and  business  contexts. 


Research  questions  associated  with  this  node  include 


D-20 

How  can  we  best  formulate  process  needs  related  to  business 
goals? 

D-21 

How  should  value-based  processes  be  deployed? 

D-22 

How  can  we  best  construct  sets  of  processes  that  when  deployed 
will  deliver  desired  business  outcomes? 
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D-23 

How  do  we  express  process  needs  so  as  to  ensure  that  an 
organization  is  able  to  adopt  processes  meeting  its  needs,  e.g.,  by 
taking  into  account  the  relationship  between  process  capability 
and  organizational  maturity?  (Depending  on  an  organization's 
process  maturity--and  many  other  factors— it  may  be  able  to 
adopt  some  processes  better  than  others.  This  needs  to  be  taken 
into  account  when  selecting  and  tailoring  processes,  i.e.,  these 
constraints  need  to  be  expressed  when  stating  an  organization's 
process  needs.) 

D-24 

How  do  we  express  process  needs  so  as  to  ensure  that  deployed 
processes  will  be  able  to  respond  effectively  to  changes  in  the 
organization  and/or  the  business  environment? 

D-25 

Do  current  models  for  process  capability  and  selection  reflect  the 
linkage  to  risk  appetite  and  risk  tolerance  in  the  organization  or 
project? 

D-26 

What  are  enablers  and  barriers  for  process  adoption? 

D-27 

How  does  the  level  of  team  competency  affect  the  deployment  of 
processes? 

D-28 

How  do  we  ensure  that  required  competencies  can  be  rapidly 
developed  and  applied  in  the  deployed  processes? 

D-29 

What  are  the  key  factors  influencing  the  successful  deployment  of 
engineered  processes? 

D-30 

How  do  the  processes  for  expressing  deployment  requirements 
affect  the  need  for  transition  infrastructure?  Are  some  processes 
cheaper  or  faster  than  others?  Do  some  require  less  infrastructure 
and,  say,  more  active  participation  by  those  who  are  affected  by 
the  changes? 

6.5.5  Research  Node  D.3.2:  Supporting  Effective 
Adoption 

The  objective  of  process  research  under  this  node  is  to  improve  the 
procedures  used  to  deploy  selected  processes  in  an  organization  so  that 
adoption  spreads  with  appropriate  speed  to  all  appropriate  structures 
and  roles  within  an  organization. 

This  node  deals  with  research  needed  on  methods  for  helping  organiza¬ 
tions  adopt  new  processes.  Such  research  includes  research  on  deter¬ 
mining  which  adoption  support  processes  are  most  effective  in  different 
organizational  circumstances. 
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D-31  How  do  we  ensure  that  required  competencies  can  be  rapidly 
developed  and  applied  in  the  deployed  processes  (JIT  Training)? 

D-32  How  can  we  support  team  performance  versus  individual 
performance? 

D-33  What  is  needed  to  develop  effective  teams  and  team  architecture 
(e.g.,  leadership  and  team  motivation)? 

D-34  How  do  you  make  the  transition  mechanisms  of  processes 

designed  for  one  context  easily  and  efficiently  adaptable  for  a 
new  context?  (i.e.,  when  an  organization  is  acquired  into  another 
enterprise)? 


D-35  How  can  CEOs  and  other  management  personnel  be  educated  and 
trained  for  process  deployment,  and  how  can  we  best  measure  the 
effectiveness  of  training? 

In  fact,  does  it  matter  how  effective  executive  sponsorship  is? 


6.5.6  Research  Node  D.3.3:  Assessing  Effectiveness 

The  objective  of  process  research  under  this  node  is  to  improve  the 
procedures  used  to  determine  the  effectiveness  of  deployment  processes 
and  deployed  processes. 

The  objective  of  process  research  under  this  node  is  to  establish  a  sound 
theoretical  and  empirical  basis  for  deployment  evaluation  and  assess¬ 
ment.  The  knowledge  gained  from  the  experience  of  process  deploy¬ 
ment  is  routinely  captured,  analyzed  and  used. 

The  work  of  this  activity  results  in  assessment  criteria  and  identified 
deficiencies,  recommended  improvements,  strengths,  and  weaknesses 
in:  the  deployment  process,  the  processes  that  were  deployed,  the  stated 
process  needs,  and  the  effectiveness  of  the  deployment. 

Research  questions  associated  with  this  node  include: 

Assess  the  Deployment  Process 

D-36  How  do  we  define  measures  of  breadth  of  process  adoption  in 
a  particular  context? 

D-37  How  do  we  define  measures  of  depth  of  process  adoption  in  a 
particular  context? 
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Assess  Adopted  Process  Effectiveness 


D-38 

What  are  the  appropriate  success  criteria  upon  which  to  judge 

deployed  processes? 

•  Can  we  develop  a  generally  accepted  set  of  measures 
describing  the  effectiveness  of  deployed  processes  and  the 
impact  of  improvement  actions? 

•  Process  reality  is  different  for  different  organizations;  how  does 
this  affect  our  understanding  of  success  criteria? 

•  What  is  the  difference  between  documented  and  practiced 
processes?  Is  the  development  performed  according  to  the 
documented  processes  or  how  does  the  actual  development 
differ  from  the  documented  process  within  an  organization? 

D-39 

What  level  of  automation  can  be  introduced  to  facilitate  the 
assessment  process? 

D-40 

How  can  we  best  validate  processes,  to  confirm  fitness  for  purpose 
and  return  on  investment? 

D-41 

What  aspects  of  process  capability  are  important  and  relevant  for 
assessment? 

D-42 

How  can  we  be  sure  that  the  results  of  process  evaluation  really 
reflect  the  ability  to  satisfy  business  needs? 

D-43 

What  lessons  can  be  learned  from  different  forms  of  process 
evaluation? 

D-44 

How  can  we  improve  the  efficiency  and  effectiveness  of  process 
evaluation  approaches? 

D-45 

How  do  we  convert  process  evaluation  to  a  continuous  activity  that 
can  be  effectively  automated? 

6.5.7  Research  Node  D.4:  Managing  Ongoing  Process 
Deployment 

Activities  under  this  node  develop:  goals  and  targets  for  process  perfor¬ 
mance  and  capability,  measures  for  monitoring  performance  and  capa¬ 
bility,  analyses  of  collected  data,  identification  of  deficiencies  between 
actual  and  targeted  performance/capability,  and  actions  to  address  these 
deficiencies  (e.g.,  corrective  actions  and  preventive  actions). 

This  node  deals  with  the  research  needed  to  ensure  that  the  ongoing  de¬ 
ployment  of  processes  in  the  organization  can  be  effectively  monitored 
and  controlled,  and  that  feedback  on  process  performance  will  reinforce 
continuing  motivation  and  sponsorship. 
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The  objective  of  process  research  under  this  node  is  to  establish  a  sound 
theoretical  framework  that  relates  the  results  of  process  implementation 
to  the  process  model  being  deployed.  A  sound  theoretical  and  empirical 
basis  exists  for  quantitative  management  of  engineering,  management 
and  improvement  processes. 


Part  of  understanding  the  success  of  a  process  deployment  must  come 
from  process  appraisal.  This  aspect  is  also  considered  in  this  node. 


D-46 

Since  all  measurement  is  taken  in  a  context,  what  is  the  best  way 
to  characterize  the  adoption  context?  What  is  the  best  way  to 
measure  that  context? 

D-47 

How  can  adoption  effectiveness  be  compared  across  organizations 
or  between  any  two  organizations?  That  is,  how  do  we  know 
which  aspects  of  deployment  to  transfer  from  one  organization  to 
another?  How  do  we  know  what  we  are  supposed  to  learn  from 
each  other? 

D-48 

What  is  the  impact  of  changes  in  the  organization  and  the 
business  environment  on  the  process  needs  and  capabilities  of  the 
organization? 

D-49 

How  do  we  best  monitor  on  a  continuous  basis  the  capabilities 
and  everyday  use  of  deployed  processes? 

D-50 

How  can  we  best  monitor  the  ongoing  returns  on  investment 
on  process  deployment  and  improvement?  Is  ROI  an  important 
indicator? 

D-51 

What  is  the  most  effective  way  of  identifying  and  addressing 
changes  in  competencies  driven  by  changes  in  the  organization 
and  business  context? 

D-52 

How  can  we  identify  causes  of  variations  in  performance  and 
capability  so  as  to  initiate  effective  process  evaluation? 

D-53 

How  do  we  create  easy  to  use  "experience  bases"  that  allow 
knowledge  to  be  stored,  updated,  and  accessed  by  developers  at 
varying  levels? 

D-54 

How  can  concepts  like  network-centric  knowledge  management 
(could  be  social  and/or  infrastructure)  be  employed  to  leverage 
process  knowledge  across  multiple  contexts? 

D-55 

What  kind  of  ontology  would  be  useful  for  classifying  process- 
based  knowledge  and  the  context  of  its  acquisition  and  use? 

D-56 

How  can  we  improve  the  design  and  use  of  process  asset 
repositories  to  support  effective  learning  from  experience? 
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D-57 

Can  the  full  validity  and  usefulness  of  statistical  process  control 
be  demonstrated  in  relation  to  software  and  systems  engineering 
and  management  processes?  If  yes,  how  should  it  be  used  to  aid 
adoption? 

D-58 

What  are  the  limitations  of  applicability  of  statistical  process 
control  in  this  context? 

D-59 

Can  the  collection  and  analysis  of  process  metrics  be  improved  to 
reduce  effort  and  increase  acceptance? 

D-60 

How  can  we  best  identify  gaps  in  the  extent  to  which  an 
organization's  processes  address  its  goals,  and  what  is  the  optimal 
mechanism  to  address  these  gaps? 
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Process  Effects  of 
Emerging  Technology 
Trends 


The  content  for  this  section  was  developed  by  Suzanne 
Garcia,  Eileen  Forrester,  John  Goodenough,  Khaled 
El-Emam,  David  Raffo,  Lynn  Penn,  and  Alan  Lawson,^ 
Additional  contributions  were  made  by  John  Morley. 
This  group  considered  all  the  emerging  technologies 
and  architectures  identified  through  IPRC  workshops. 


Research  Areas  and  Questions  in  this  Section 

•  Technology  Factor: 

Continuous  Requirements  Evolution 
(T1-T23) 

•  Technology  Factor: 

Incomplete  Knowledge 
(T24-T37) 

K 

•  Technology  Factor: 

Heterogeneous  Component-Based  Systems  Integration 
(T38-T56) 
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7.1  Introduction 


Can  anyone  define  the  technologies  that  will  influence  or,  perhaps, 
guide  our  future?  Does  anyone  know  the  effects  a  new  technology  will 
have  on  product  qualities  or  process?  IPRC  members  delved  into  specu¬ 
lation  about  the  ways  in  which  new  technologies  might  affect  the  pro¬ 
cesses  we  use — witness  the  scenarios  of  the  future  described  in  Section 
9  and  the  definition  of  driving  forces  in  Section  2.2. 

At  the  conclusion  of  that  speculation  and  definition,  however,  what  we 
have  confirmed  is  that  we  cannot  know  with  certainty  what  will  be.  We 
cannot  project  today’s  technologies  into  the  future  with  any  acceptable 
accuracy,  because  we  cannot  fully  predict  the  course  an  innovation  will 
take  as  it  moves  from  concept  through  research  to  use.  Likewise,  we  are 
hard-pressed  to  explicitly  predict  tomorrow’s  technologies  because  we 
cannot  influence  many  of  the  forces  that  will  cause  them  to  be  developed. 

Yet  despite  those  limitations,  we  do  know  something  valuable  about 
the  trends  we  see  with  technology:  we  can  identify  attributes  of  tech¬ 
nology  that  have  significant  impact  on  processes  within  a  system  life 
cycle.  Because  they  affect  a  system  or  product’s  life  cycle,  those  at¬ 
tributes — we  call  them  technology  factors — are  important  for  process 
research.  In  this  section,  we  have  delineated  three  overarching  factors 
that  emerged  from  our  considerations  of  the  varying  possible  technolo¬ 
gies  of  the  future: 

1 .  Continuous  Requirements  Evolution 

We  must  acknowledge  that,  for  many  system  types  of  the  future, 
we  will  need  to  initiate,  manage,  and  evolve  sets  of  necessarily 
incomplete  requirements  over  multiple  iterations  of  system 
evolution  and  reconfiguration. 

2.  Incomplete  Knowledge 

Beyond  continuous  evolution  and  therefore  incomplete 
knowledge  about  requirements,  many  systems  will  be  evolving 
the  technologies  in  use,  and  the  stakeholders  that  are  participating 
in  evolution,  at  the  same  time  that  they  are  trying  to  deliver  a 
usable  system.  This  creates  a  development  context  of  necessarily 
incomplete  knowledge  for  a  major  part  of  a  system’s  overall  life. 

3.  Heterogeneous  Component-Based  Systems  Integration 
We  are  moving  away  from  hand-crafted,  custom-developed 
solutions  into  a  world  of  heterogeneous,  component-based 
systems  (systems  of  systems).  The  integration  challenge  in  this 
environment  extends  beyond  the  technical  integration  (though 
that  will  be  difficult  enough)  into  the  socio-technical  aspects  of 
process  integration  in  particular. 
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If  this  sounds  like  Fred  Brooks  Jr.’s  “No  silver  bullet,”  well  it  is!  In 
that  famous  paper,  Brooks  drew  attention  to  the  inherent  complexity 
of  software.  Here,  we  are  simply  observing  that  it  is  increasing,  even 
though  we  barely  could  get  intellectual  control  of  the  previous  level 
of  complexity.  A  recent  observation  by  William  Wulf,  president  of  the 
U.S.  National  Academy  of  Engineering,  is  instructive.  In  an  address 
at  the  convocation  of  the  University  of  Southern  California  Center  for 
Systems  and  Software  Engineering  on  October  24,  2006,  Wulf  said 

The  fact  that  physical  systems  are  described  by  continuous 
mathematics  is  what  enables  thorough  testing.  It  is  possible 
to  test  a  physical  system  for  one  input  value  and  know  that  its 
behavior  for  small  changes  in  the  input  will  be  very  similar 
Thus,  if  one  tests  a  range  of  input  values  that  are  ‘ close 
enough,  ' you  have  exhaustively  tested  the  system. 

Not  so  for  software.  If  one  of  the  1010  bits  of  my  laptop  's  memory 
is  changed,  a  large,  discontinuous  change  in  behavior  may 
occur.  That,  in  turn,  means  that  an  ‘exhaustive'  test  of  a  piece 
of  software  must  consider  all  possible  ‘states'  of  memory. 

Just  to  get  a  feel  for  what  that  means,  each  bit  in  memory  can 
be  in  one  of  two  states  -  so  there  are  2**1010  (roughly  10**108) 
of  them.  That's  10  raised  to  a  power  that  has  a  one  followed 
by  eight  zeros.  Just  for  comparison,  I  believe  there  are  about 
10100,  atoms  in  the  universe.  Ten  raised  to  a  power  of  1  followed 
by  two  zeros.  Thus,  the  number  of  atoms  in  the  universe  is  an 
infinitesimal  fraction  of  the  number  of  states  in  my  laptop 's 
memory. 

Bottom  line,  except  for  trivial  programs,  exhaustive  testing  of 
software  is  impossible.  The  lack  of  mathematical  continuity  is 
the  culprit. 
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The  purpose  of  our  discussion  of  these  factors  is  to  propose  an  approach 
to  forming  research  questions  while  acknowledging  that  the  forces  af¬ 
fecting  process  development  for  any  particular  environment  may  be 
unknown  or  not  clearly  understood.  These  are  not  the  only  factors  that 
were  considered,  and  it  is  easily  conceivable  that  other  factors  will  in¬ 
crease  in  importance  to  processes  for  system  construction  and  evolution 
over  time.  Future  IPRC  content  is  likely  to  address  adding  additional 
technology  factors. 

In  process  terms,  systems  and  software  engineering  has  many  prob¬ 
lems  for  which  adequate  solutions  have  not  been  found.  The  research 
questions  presented  in  earlier  sections  of  this  report  provide  testament 
to  this  fact.  To  be  successful,  any  new  technology  or  architecture  will 
need  either  to  solve  an  existing  problem  or  open  up  new  opportunities. 
Industry  trends  suggest  that 

•  Systems  are  becoming  ever  more  complex,  both  in  terms  of  size  and 
architecture — often  involving  a  multidisciplinary  approach — and 
reuse  of  open  source  components  is  increasing,  along  with  the  trend 
towards  the  dynamic  trading  of  services  between  interoperating 
technological  systems.  This  leads  to  systems  that  are  built  from 
heterogeneous  components,  often  with  diverse  stakeholders.  We 
characterize  this  as  “heterogeneous  component-based  integration.” 

•  There  is  an  increasing  need  to  cope  with  very  dynamic  environments 
where  systems  are  regularly  upgraded  to  accommodate  continually 
changing  business  requirements.  We  struggle  with  evaluating  the 
impact  of  these  changes,  both  on  the  systems  themselves  and  on  the 
environments  into  which  these  systems  are  placed.  We  characterize 
this  as  a  state  of  “continuous  requirements  evolution.” 

•  As  we  move  closer  and  closer  to  a  virtual  world,  trust  and  assurance 
of  system  integrity  is  becoming  a  major  concern,  particularly  where 
biological  linkages  exist.  In  these  contexts,  we  are  often  faced  with 
the  dilemma  of  the  need  for  high  trust  and  assurance,  while  having 
incomplete  knowledge  of  the  constituents  and  the  system’s  state. 

A  complicating  factor  is  that  more  and  more  non  software 
engineering  professionals  are  building  systems,  taking  us  closer 
to  a  merger  of  the  software  engineering  discipline  with  the 
target  domains  themselves.  However,  this  is  not  yet  providing 
an  integration  of  software  development  processes  with  domain- 
specific  processes.  We  characterize  this  is  as  working  in  a  state  of 
“incomplete  knowledge.” 

•  We  operate  in  an  increasing  global  domain,  both  for  the  procurement 
and  provision  of  technological  systems.  This  is  one  of  the  underlying 
causes  of  the  heterogeneity  that  continues  to  increase  in  our 
systems  and  systems  of  systems.  We  treat  this  in  our  section  on 
“heterogeneous  component-based  integration.” 
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Figure  5  provides  a  framing  of  three  dimensions  that  we  believe  have  T 

impact  when  looking  at  how  emerging  technologies  affect  processes  of  " 

system  construction  and  evolution.  As  the  degree  of  dynamism  increas¬ 
es  in  the  system  state  (e.g.,  constituents  coming  in  and  out  of  systems 
of  systems),  we  also  see  a  decrease  in  the  degree  of  knowledge  we  can 
have  about  the  system’s  state.15  This  might  not  be  as  much  of  a  process 
issue  if  we  weren’t  interested  in  greater  and  greater  degrees  of  assur¬ 
ance  for  the  systems  that  we  construct,  regardless  of  their  conditions  of 
construction.  This  tension  among  dynamism,  knowledge,  and  desire  for 
assurance  underpins  the  three  factors  that  we  highlight  in  this  section. 

The  tension  among 
dynamism,  knowledge, 
and  desire  for  assurance 
underpins  the  three  factors 
that  we  highlight  in  this 
section. 
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Figure  5:  The  Trend  Toward  Decreasing  System  Knowledge 

These  issues  are  also  connected  to  the  research  themes  presented  earlier. 
In  general,  however,  we  can  see  that  there  are  three  broad  areas  where 
more  research  is  required.  These  are  concerned  with  how  to  handle 
(a)  continuously  changing  requirements,  (b)  circumstances  in  which  we 
necessarily  have  incomplete  knowledge,  and  (c)  the  world  of  large-scale 
interoperating  technological  systems. 


15  Position  of  systems  in  Figure  5  are  meant  to  indicate  only  notional  relationships 
between  dynamism  and  knowledge. 
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7.2  Method 


We  initially  looked  at  technologies  and  what  is  driving  their  develop¬ 
ment  in  terms  of  a  matrix  (see  Figure  6).  This  approach  is  concerned 
with  the  effects  that  new  and  emerging  technologies  related  to  informa¬ 
tion  and  communications  technology  might  have  on  the  processes  used 
to  conceive,  build,  deploy,  operate,  and  evolve  products  and  services 
that  make  use  of  them.  In  analyzing  the  many  possibilities  that  may 
come  up  in  the  future,  the  approach  has  been  resolved  into  three  tech¬ 
nology  factors  that  cut  across  multiple  technological  possibilities  and 
have  significant  process  research  implications.  We  examined  these  fac¬ 
tors,  rather  than  trying  to  build  maps  for  specific  technologies,  because 
the  process  issues  that  arise  for  individual  technologies  will  be  more 
general  than  specific. 

There  will  be  exceptions  to  this  statement;  however,  in  looking  at  a 
process  research  framework,  we  need  to  deal  with  the  general  process 
issues  that  will  be  present  in  many  of  the  emerging  technologies,  since 
they  are  the  high  leverage  points.  Users  of  the  framework  may  find  it 
useful  to  think  about  how  the  questions  below  modify  some  of  the  re¬ 
search  questions  in  the  four  core  process  research  themes  (see  Sections 
3-6)  if  both  the  technology  factor  and  the  considerations  of  a  particular 
research  node  are  in  play.  See  the  Appendix  for  an  example  of  consider¬ 
ing  technology  factors  and  other  research  nodes  together. 

In  the  matrix  shown  in  Figure  6,  at  the  intersection  of  challenges  and 
system  types,  we  see  some  areas  emerge  in  relief  (see  gray  cells  in 
Figure  6),  such  as  the  high  acceleration  of  importance  that  complexity, 
dynamism,  and  emergent  behavior  have  in  a  systems  area  that  is  still 
early  in  its  evolution — self-organizing  systems. 

New  technologies  are  highly  likely  to  imply  new  process  needs/capa- 
bilities.  One  of  the  interesting  process  aspects  about  new  technologies  is 
that  we  are  typically  trying  to  learn  about  them  as  we  build  them.  This 
leads  to  a  different  learning  process  than  is  present  later  in  their  evolu¬ 
tion.  This  difference  will  be  reflected  in  process  objectives  and  research 
questions  that  may  be  specific  to  a  particular  system  evolution  stage. 

As  an  emerging  technology  factor  is  exercised  from  today’s  process 
capability  to  the  future,  depending  on  the  characteristics  of  the  technol¬ 
ogy,  different  process  capabilities  may  be  needed  to  succeed.  The  team 
created  process  objectives  and  research  questions  that  cover  a  wide 
spectrum  of  system  evolution  contexts.  Research  users  may  wish  to 
confine  their  process  research  focus  to  a  narrower  context  than  is  ex¬ 
pressed  in  these  questions. 
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Figure  6:  Process  Challenges 


As  a  reminder,  there  are  three  technology  factors  encompassed 
in  this  section: 

1 .  Continuous  Requirements  Evolution 

2.  Incomplete  Knowledge 

3.  Heterogeneous  Component-Based  Systems  Integration 

7.3  Technology  Factor:  Continuous 
Requirements  Evolution 

The  process  objectives  for  this  node  are  to  initiate,  manage,  and  evolve 
sets  of  necessarily  incomplete  requirements  over  multiple  iterations  of 
system  evolution  and  re-configuration. 

Whether  it  is  the  evolution  of  a  system  of  systems  or  the  integration  of 
nanotechnology  into  the  latest  soft  body  armor,  looking  at  requirements 
from  anything  except  a  continuous  viewpoint  will  be  less  and  less  fea¬ 
sible,  especially  where  emerging  technologies  are  in  play. 

7.3.1  Characterizing  the  Current  State  of  the  Practice 

Requirements  changes  within  software  that  is  specifically  produced  for 
limited  users  is  a  reality  today  and  without  constant  attention  and  pro¬ 
cesses  which  support  such  an  evolution,  industry  is  often  paralyzed  and 
unable  to  meet  the  needs  of  the  customer  thus  destined  to  fail.  Although 
requirements  are  supposedly  stabilized  within  the  first  phase  of  the  life 
cycle,  software  methodologies,  like  agile,  are  calling  for  real  time  de¬ 
velopment  and  acceptance  of  requirements  throughout  a  highly  iterative 
product  development  life  cycle. 

The  level  of  expected  dynamism  in  requirements  is  not  usually  ac¬ 
counted  for  in  life  cycles  in  prevalent  use  today.  It  is  a  fact  that  require¬ 
ments  “popping  in”  are  not  handled  well.  We  go  out  of  our  way  to  drive 
dynamism  out  of  the  life-cycle  stages  beyond  requirements  analysis  for 
large,  monolithic  systems  of  systems. 

Although  methods  with  an  emphasis  on  agility  and  flexibility  are  in 
some  use  today,  there  is  considerable  variation  in  how  these  methods 
are  practiced.  Agilists  who  have  surveyed  conformance  of  project  prac¬ 
tices  to  the  principles  and  practices  of  particular  methods  have  found 
that  the  agile  terms  are  often  used  without  applying  the  actual  practices 
in  the  way  they  are  intended.16 

We  try  to  minimize  risk  by  minimizing  variability.  We  go  out  of  our 
way  to  drive  dynamism  out  of  the  life-cycle  stages  beyond  requirements 
analysis  for  large  monolithic/sy stems  of  systems,  leading  to  overly  con¬ 
strained  systems  that  are  unable  to  evolve  as  needed.  Synchronization 

16  These  insights  are  taken  from  a  private  conversation  of  Suzanne  Garcia  and  Alistair 
Cockburn  (developer  of  the  Crystal  Methods  approach)  in  September  2006. 
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and  management  of  stakeholder  expectations  with  system  require¬ 
ments  volatility  is  problematic.  When  we  do  deal  with  changes,  we 
often  limit  our  analysis  of  the  change  impact  to  local  effects.  This 
often  leads  to  missing  important  system-level  effects  of  local  changes. 
Release  cycles  are  ad  hoc.  We  have  little  visibility  into  planned  re¬ 
leases  based  on  requirements.  Systems  are  often  looked  at  singularly 
versus  part  of  the  bigger  picture. 

7.3.2  Characterizing  the  Desired  State  of  the  Practice 

In  a  future  where  we  explicitly  account  for  and  embrace  continuous 
requirements  evolution,  we  would  have  a  set  of  models,  analysis 
techniques,  and  methods  that  allow  us  to  characterize  requirements 
dynamism  both  for  build  purposes  and  prediction  purposes.  We  would 
have  a  set  of  validated  (or  at  least  useful!)  metrics  for  characterizing 
the  dynamism  of  a  requirements  set  in  its  initial  state.  Using  analyses 
and  models  to  guide  creation  of  the  initial  version  of  the  system, 
we  would  be  able  to  characterize  it  for  adaptation  and  assurance 
purposes.  We  could  effectively  build  an  initial  version  of  a  system  that 
is  adaptable  to  a  continuously  evolving  set  of  requirements.  We  would 
be  able  to  characterize  the  level  of  known  requirements  satisfaction 
with  an  initial  system  build,  and  the  level  of  dynamism  expected  in  its 
future.  One  of  the  ways  we  would  support  evolution  is  by  minimizing 
the  constraints  in  the  initial  system  so  that  the  solution  space  for 
future  versions  would  not  be  unduly  constrained. 

To  achieve  comfort  with  continuous  requirements  evolution,  we 
would  use  modeling  and  simulation  much  more  extensively,  and  these 
models  would  effectively  express  the  relevant  elements  of  multiple 
systems  so  as  to  be  able  to  visualize  change  impacts  across  the 
different  systems.  These  models  would  also  effectively  express  the 
uncertainties  and  probabilistic  aspects  of  requirements.  When  dealing 
in  system  of  systems  contexts,  architectures  would  be  designed  to 
permit  effective  analysis  across  multiple  system  nodes,  allowing  us 
to: 

•  anticipate  indirect  and  cascading  effects  due  to  different  process 
choices 

•  predict  the  effects  of  system  node  interchange 

•  determine  risk/reward  ratios  for  different  system-of-systems 
configurations 

All  of  this  would  allow  the  impacts  of  changes  to  be  modeled  success¬ 
fully  across  multiple  system-of-systems  elements  in  a  timely  manner. 

Research  questions  associated  with  this  technology  factor  include17 

17  The  questions  associated  with  the  technology  factors  in  this  section  are  identi¬ 
fied  with  the  letter  T.  Elsewhere  in  this  report,  research  questions  for  the  theme  on  the 
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T-1 

How  do  you  anticipate  the  pressures  for  change  early  in  the 
system  life? 

T-2 

How  do  you  anticipate  requirements  volatility  and  unknown, 
unidentified  requirements  early  in  the  development  life  cycle?  How 
do  you  adapt  processes  to  allow  for  this? 

T-3 

What  metrics  can  we  develop  to  characterize  the  level  of  known 
requirements  satisfaction  and  the  level  of  dynamism  expected  in 
future? 

T-4 

How  to  you  characterize  the  level  of  known  requirements 
satisfaction  with  initial  build  and  the  level  of  dynamism  expected 
in  future? 

T-5 

How  do  you  support  a  requirements  management  system/process 
that  accounts  for  both  expected  and  unexpected  change? 

T-6 

How  do  you  instantiate  a  life  cycle  that  explicitly  recognizes 
incomplete  requirements  as  its  basis  for  initial  development? 

T-7 

How  do  you  instantiate  development  with  partial  requirements? 

T-8 

How  do  we  instantiate  processes  for  verification  and  validation  in 
initial  build  that  account  for  high  requirements  dynamism? 

T-9 

How  do  you  keep  verification  and  validation  tightly  coupled  to 
requirements  when  the  requirements  are  "a  moving  target"? 

T-10 

How  do  you  gather  and  filter  the  relevant  data  to  support  "the  next 
evolution"  in  your  requirements? 

T-11 

How  do  you  instrument  the  system  to  collect  information  that  will 
permit  appropriate  requirements  evolution? 

T-1 2 

How  do  you  model  the  eco-system  the  system  will  be  a  part  of? 

T-1 3 

How  do  you  measure  requirements  volatility  in  a  useful  way? 

T-1 4 

How  do  you  sustain  and  evolve  an  eco-system  model  for  the  world 
the  system  lives  in? 

T-1 5 

What  is  important  and  essential  to  include  in  SoS  requirements 
analysis  models? 

T-1 6 

What  data  about  user  behavior  needs  to  be  collected  that  will  help 
infer  the  need  for  requirement  changes? 

T-1 7 

What  information  about  new  technologies  needs  to  be  collected  to 
help  infer  the  need  for  requirements  changes  and  the  impact  of  the 
new  technology  on  the  existing  system 

relationships  between  proecsses  and  product  qualities  are  identified  with  a  Q,  those  for  the 
process  engineering  theme  with  an  E,  for  managing  process  processes  with  a  P,  and  for 
process  deployment,  a  D.  The  questions  shown  in  the  Appendix  are  labeled  with  an  S. 
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T-18 

What  replaces  "quality  =  conformance  to  requirements"  in  a 
system  where  continuous  requirements  evolution  is  present? 

T-19 

How  do  you  instantiate  an  assurance  strategy  (a  process  for  doing 
so)  given  that  a  subset  of  the  requirements  do  not  have  to  be  met 
all  the  time?  (i.e.,  in  a  SLA  context) 

T-20 

How  do  you  establish  a  release  cycle  that  balances  environment 
change  with  user  ability  to  integrate  or  deploy  the  new  increment? 

T-21 

How  do  you  anticipate  (determine)  the  pressures  for  change  at  a 
given  point  in  the  system  life? 

T-22 

What  are  the  relevant  utility  functions  for  establishing  release 
cycles? 

T-23 

How  do  you  recognize  when  to  re-architect  for  future  anticipated 
changes? 

7.4  Technology  Factor:  Incomplete 
Knowledge 

The  process  objectives  for  this  node  are  (i)  to  enable  systems  develop¬ 
ers  to  make  decisions  effectively  and  efficiently  throughout  the  life  of  a 
system  while  accommodating  necessarily  incomplete  or  uncertain  sys¬ 
tem  knowledge  and  (ii)  to  be  able  to  define  acceptable  levels  of  incom¬ 
pleteness  or  uncertainty  for  developers  in  different  system  contexts. 

Many  of  the  technologies  and  architectures  that  are  emerging  (e.g., 
net-centric  operations),  operate  in,  and  therefore  are  built  in,  a  state  of 
explicitly  incomplete  knowledge — about  the  context  of  use,  the  other 
constituents  that  are  part  of  the  product’s  operational  context,  the  effects 
of  the  other  organizations  involved  in  a  development  effort,  system 
state,  etc.  This  factor  is  particularly  urgent  to  address  in  systems  where 
failure  would  have  severe  negative  consequences,  so  the  real  issue  ad¬ 
dressed  is  the  incomplete  knowledge  in  a  state  of  continuing  high  need 
for  performance,  safety,  security,  etc. 

While  our  focus  here  is  on  the  knowledge  of  the  developer  of  the  system, 
another  entire  topic  is  the  incomplete  knowledge  and  skill  of  the  user. 

For  example,  it  is  typical  that  as  technologies  mature  there  is  pressure  to 
“de-skill”  their  use.  The  de-skilling  is  sometimes  communicated  as  “self- 
service.”  De-skilling  implies  less  knowledge  than  those  who  hitherto 
interfaced  to  or  used  technology  (think  of  the  help  you  get  at  call  centers 
today  regarding  answers  to  technical  questions),  so  there  is  an  immediate 
implication  that  users  of  technology  will  know  less,  will  have  incomplete 
knowledge.  Sometimes  the  de-skilling  is  liberating  and  makes  technology 
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available  to  “the  rest  of  us,”  and  sometimes  it  adds  a  level  of  complexity 
that  we  as  professional  systems  developers  cannot  yet  cope  with  (think  of 
trying  to  de- skill  nuclear  power  plant  operations). 

7.4.1  Characterizing  the  Current  State  of  the  Practice 

For  system  contexts  in  which  incomplete  knowledge  is  (or  will  be)  preva¬ 
lent,  today’s  component-  and  system-build  approaches  are  a  poor  fit,  be¬ 
cause  they  force  systems  to  drive  to  certainty.  We  often  push  for  certainty 
before  understanding  an  adequate  set  of  contexts — for  use,  for  test  beds, 
or  for  configurations.  We  do  not  know  what  data  is  safe  to  ignore  or  sim¬ 
plify  in  different  configurations  or  system  states.  We  have  trouble  charac¬ 
terizing  how  much  data  is  enough  to  collect  in  relation  to  making  process 
and  other  choices.  Processes  that  help  to  assess  the  probability  that  what 
is  “known”  is  actually  “true”  are  not  in  widespread  use. 

In  addition,  when  faced  with  a  set  of  data  or  an  analysis  result,  tech¬ 
niques  we  currently  use  are  insufficient  for  characterizing  the  complete¬ 
ness  of  our  knowledge  about  the  data.  Many  of  the  standard  analysis 
techniques  (e.g.,  FMEA,  the  failure  modes  and  effects  analysis)  do  not 
translate  well  into  an  environment  of  incomplete  system  knowledge 
(i.e.,  where  you  do  not  know  if  the  component  at  the  end  of  a  fault  tree 
is  actually  working  or  not  working).  Quantitative  assurance  with  an  ex¬ 
plicit  probabilistic  focus  is  not  an  accepted  mode  in  the  test  and  evalu¬ 
ation  community,  even  though  it’s  a  necessary  mode  in  systems  of  high 
uncertainty,  especially  where  there  is  minimum  tolerance  for  failure. 
Therefore,  release  processes  based  on  quantitative  assurance  are  neither 
sufficient  nor  adequately  used. 

7.4.2  Characterizing  the  Desired  State  of  the  Practice 

When  we  have  systems  that  are  characterized  by  continuing,  incomplete 
knowledge,  we  would  use  system  building  approaches  that  effectively 
deal  with  low  probability  but  high  impact  risks.  We  would  consistently 
use  processes  that  identify  clearly  the  boundaries  of  complete  and  in¬ 
complete  knowledge  about  operational  scenarios,  system  state,  and  sys¬ 
tem  configuration.  Approaches  would  be  well  understood  and  used  for 
collecting  the  “right”  data  (that  which  cannot  be  ignored  or  simplified) 
in  different  operational  contexts.  Quantitative  assurance  with  an  explicit 
probabilistic  focus  would  be  an  accepted  mode  in  the  test  and  evalua¬ 
tion  community.  When  making  release  decisions,  we  would  use  release 
processes  based  on  quantitative  assurance  results.  We  would  also  have 
processes  for  managing  configurations  that  effectively  deal  with  un¬ 
bounded  and  incomplete  contexts. 

Overall,  we  would  have  processes  that  drive  to  certainty  only  when  the 
time  is  right,  and  the  knowledge  needed  is  available.  And  we  would 
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cause  we  have  processes  that  appropriately  handle  them. 
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Research  questions  associated  with  this  technology  factor  include 


T-24 

What  are  key  elements  of  the  design  or  architecture  that 
accommodate  operating  and  evolving  a  system  when  there  is,  and 
will  continue  to  be,  incomplete  system  state  knowledge? 

T-25 

For  a  given  system  type  and  operational  context,  can  we 
characterize  a  framework  of  where  and  how  incomplete 
knowledge  can  be  tolerated  or  dealt  with? 

T-26 

What  are  the  processes  for  determining  what  is  important  to  know 
or  not  know  about  system  state  and  configuration? 

T-27 

What  are  the  processes  for  using  knowledge  (complete  or 
incomplete)  about  system  state  and  configuration  for  system  build 
and  integration? 

T-28 

What  are  the  processes  for  determining  what  is  knowable  and 
unknowable  about  the  operational  state  or  configuration  of  a 
system  at  different  points  in  its  evolution? 

T-29 

Flow  do  we  determine  what  data  will  help  us  to  move  from 
"unknown  unknowns"  to  "known  unknowns"  to  "knowns"? 

T-30 

Flow  do  we  determine  the  "shelf  life"  of  data  to  be  collected  at 
various  points  in  system  evolution? 

T-31 

What  are  approaches  for  understanding  which  elements  of  your 
configuration  can  be  ignored  at  any  particular  time? 

T-32 

Flow  do  we  provide  frameworks  for  eliciting  more  complete 
pictures  of  system  context  and  data  more  effectively? 

T-33 

Flow  do  we  effectively  collect  information  about  ambiguous  data 
and  data  types  to  help  reduce  that  ambiguity? 

T-34 

What  kinds  of  modeling  and  analysis  techniques  for  assurance 
help  us  to  characterize  the  level  of  completeness  of  our  assurance 
strategy  in  relation  to  the  knowledge  that  is  available? 

T-35 

Flow  do  we  provide  "data  cleansing"  to  deal  with  data  issues 
when  we're  acknowledging  incomplete  data  sets? 

T-36 

Given  that  conclusions  from  assurance  activities  are  necessarily 
probabilistic  in  nature,  what's  the  decision  process  to  use  them  for 
release  decisions,  etc.? 

T-37 

What  kinds  of  processes  can  be  used  for  relevant  quantitative 
assurance  when  system  state  or  context  of  use  is  unknown? 
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7.5  Technology  Factor:  Heterogeneous 
Component-Based  Systems 
Integration 

The  overall  process  objective  for  this  node  is  to  establish  effective 
processes  for  conceiving,  orchestrating,  constructing,  deploying  and 
evolving  complex  systems  composed  from  heterogeneous  components. 
In  this  context,  we  consider  heterogeneous  components  to  include  both 
system  components  (some  of  which  may  be  autonomous)  and  the  orga¬ 
nizations  that  construct  and  integrate  them. 

Many  of  the  technologies  that  we  envisage  for  the  future  move  us 
away  from  hand-crafted,  custom-developed  solutions  into  a  world  of 
integrating  heterogeneous,  component-based  systems.  The  integration 
challenge  in  this  environment  extends  beyond  the  technical  integration 
(though  that  will  be  difficult  enough)  into  the  socio-technical  aspects  of 
process  integration  in  particular. 

Many  industries  which  have  previously  not  relied  on  software  (such  as 
household  appliances  and  automobiles)  are  becoming  increasingly  de¬ 
pendant  on  its  utility.  Their  ability  to  produce  inexpensive  products  with 
acceptable  quality  levels  is  critical  to  their  success. 

7.5.1  Characterizing  the  Current  State  of  the  Practice 

One  of  the  fallacies  of  our  current  approach  to  systems  integration  is  the 
assumption  that  high  quality,  “best  of  breed”  components  will  automati¬ 
cally  lead  to  an  easily  integrable  system  solution.  The  emerging  litera¬ 
ture  in  system  of  systems  interoperability  highlights  the  many  problems 
encountered  when  components  of  long-standing  use  individually  are 
composed  into  a  larger  system  context.  The  ultimate  source  of  this  is¬ 
sue  is  that  systems  of  systems  composed  from  heterogeneous,  often 
autonomous  constituents,18  exhibit  emergent  properties — properties  that 
are  not  present  in  the  individual  components,  but  are  exhibited  when  the 
system  operates  as  a  whole. 

When  integrating  systems  from  heterogeneous  components,  we  have 
trouble  understanding  whether  the  components  conform  to  agreed-upon 
standards,  especially  with  “private”  components  that  we  must  treat  as 
black  boxes.  We  often  do  not  even  understand  what  data  is  relevant  to 
collect  from  components  to  analyze  service  quality  across  a  system  con¬ 
text.  An  exception  to  this  is  the  open  source  movement,  where  standards 
conformance  and  other  questions  can  be  answered  more  transparently 
than  with  many  privately  built  components. 


18  By  constituents,  we  mean  both  the  organizational  elements  involved  in  the  building  of  a 
system  or  component  and  the  components  themselves. 
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When  trying  to  integrate  in  a  context  where  we’re  working  with  hetero¬ 
geneous  organizations,  we  often  find  a  lack  of  trust  among  the  constitu¬ 
ents,  increasing  the  overhead  related  to  managing  these  types  of  efforts 
for  all  concerned.  One  source  of  this  lack  of  trust  is  varying  capability 
of  engineering  and  project  management  processes  among  the  constitu¬ 
ents.  From  an  SEI  process  maturity  viewpoint,  we  have  trouble  resolv¬ 
ing  process  maturity  differences  among  the  different  constituents.  A 
frequently  cited  observation  is  that,  in  CMMI  math,  ML3+ML1=ML1. 
In  other  words,  higher  maturity  organizations  are  more  often  brought 
down  to  the  level  of  process  performance  of  their  lower  maturity  part¬ 
ners  than  the  opposite. 

When  it  comes  to  verifying  and  validating  heterogeneously  constructed 
systems,  we  do  not  know  how  to  validate  systems  against  scenarios  of 
use  that  have  not  yet  been  conceived,  and  even  if  we  could  envision  all 
the  scenarios  of  use  of  a  particular  system,  the  cost  to  test  and  evaluate 
the  entire  system  across  all  the  possible  contexts  of  use  is  prohibitive. 
Some  organizations  have  a  risk-based  approach  to  evaluating  suitability 
of  software  for  integration,  but  these  practices  are  not  widely  used.  We 
do  not  have  validated  techniques  for  analysis-based  verification  and 
validation  in  this  space  yet  (although  research  in  this  area  has  begun).  In 
general,  when  it  comes  to  verification  and  validation,  we’re  testing  the 
strength  of  the  bricks,  not  the  strength  of  the  building. 

One  of  the  benefits  of  systems  integrated  from  heterogeneous  compo¬ 
nents  is  the  potential  for  beneficial  emergent  properties  to  manifest. 
However,  we  lack  effective  methods  for  analyzing  and  characterizing 
the  emergent  properties  of  a  heterogeneously  composed  system,  with 
regard  to  both  beneficial  and  detrimental  emergent  properties.  [Fisher 
2006] 

7.5.2  Characterizing  the  Desired  State  of  the  Practice 

The  overall  desired  state  for  integrating  heterogeneous  components 
into  usable  systems  is  to  successfully  achieve  interoperability  and  other 
beneficial  quality  factors  (e.g.,  security,  privacy)  in  an  environment  of 
increasing  dynamism. 

We  would  use  interoperable  processes  effectively  for  heterogeneous 
“planned”  systems.  Integration  checklists  would  systematically  al¬ 
low  risk-based  choices  for  building  initial  heterogeneous  systems. 
“Black  box”  integration  would  be  consistently  possible  and  we  would 
treat  components  in  a  black  box  fashion  for  integration  purposes.  We 
would  understand  and  articulate  the  inherited  risks  among  components. 
Standard  processes,  interfaces,  and  technologies  would  support  consis¬ 
tently  achieving  interoperability. 
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We  would  observe  appropriate  privacy  and  security  practices  when  col¬ 
lecting  and  reporting  data;  this  would  improve  the  trust  level  of  constitu¬ 
ents  in  a  heterogeneous  system  integration.  Standards  would  be  routinely 
used  to  specify  the  data  to  be  published  by  components  expecting  to 
integrate  into  heterogeneous  systems. 

We  would  be  able  to  accurately  characterize  and  facilitate  the  manifesta¬ 
tion  of  beneficial  emergent  properties,  and  minimize  the  incidence  and 
impact  of  detrimental  emergent  properties. 


Research  questions  associated  with  this  technology  factor  include 


T-38 

Are  there  requirements  approaches  that  will  reduce 
integration  and  assurance  challenges  for  heterogeneous 
systems? 

T-39 

How  do  we  understand  the  indirect  and  cascading  effects 
that  are  inherited  from  component  systems  when  we  try  to 
initially  integrate  them? 

T-40 

How  can  we  create  market  and/or  regulatory  forces  that 
incentivize  vendors  to  create  products  that  are  verified, 
secure,  interoperable,  etc.? 

T-41 

How  do  we  change  the  education  system  to  prepare  future 
engineers  to  work  productively  in  this  environment? 

T-42 

How  do  you  decide  what  data  needs  to  be  collected  from  the 
heterogeneous  components  and  suppliers  for  acceptance  into 
integration? 

T-43 

How  do  you  build  the  trust  required  to  ensure  that  accurate 
data  is  being  passed  in  a  relevant  fashion  among  nodes  in 
the  system? 

T-44 

How  do  you  collect  and  update  data  on  system  performance 
to  establish  service  quality  of  individual  components/new 
requirements? 

T-45 

What  kinds  of  processes,  models,  techniques,  etc.  help  you 
to  analyze  which  new  standards  actually  help,  rather  than 
hinder,  heterogeneous  system  evolution? 

T-46 

How  do  you  analyze  data  collected  on  system  performance 
to  characterize  service  quality  of  individual  components  and 
new  requirements  needed? 

T-47 

How  do  we  analyze  components  to  determine  their  suitability 
for  inclusion  in  a  system  of  interest? 

T-48 

How  do  we  analyze  components  related  to  their  security, 
workflows,  user  roles,  decision  support  to  characterize  those 
aspects  across  the  system? 
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T-49 

How  do  we  analyze  the  communication  paths  and 
performance  among  the  components  to  ensure  successful — 
and  eventually  seamless — integration? 

T-50 

How  do  we  analyze  emergent  properties  (both  good  and  bad) 
that  are  present  in  the  heterogeneous  system? 

T-51 

How  do  you  establish  processes  that  can  navigate  different 
legal  environments  when  assuring  a  heterogeneous  system? 

T-52 

How  do  you  optimize  an  assurance  strategy  based  on  usage 
of  components  and  scale  of  integration  effort? 

T-53 

How  do  you  avoid  "big  bang"  integration  processes? 

T-54 

How  do  you  deal  with  asynchronous  upgrades  of  different 
elements  of  your  system? 

T-55 

In  an  environment  of  continuously  evolving  requirements, 
how  do  you  allocate  and  or  communicate  new  or  changing 
requirements  across  all  the  relevant  components? 

T-56 

How  do  you  manage  alignments  of  user  roles,  security, 
workflows,  decision  support,  etc  when  in  the  build/ 
integration  process  when  you  have  all  the  requirements 
changing? 
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Scenarios 


View  the  full  scenarios 
online — or  contribute 
your  own. 


Processes  do  not  exist  outside  of  some  context.  The  scenarios  were 
developed  to  analyze  the  impact  of  future  social,  political,  technologi¬ 
cal  trends  on  the  software,  systems,  enterprises,  and  IT.  The  scenarios 
represented  extensions  of  current  trends  both  optimistic  and  pessimistic. 
Each  scenario  was  analyzed  in  terms  of  the  implications  for  products, 
people,  projects,  management,  and  organization.  The  implications  were 
then  assessed  in  terms  of  the  requirements  they  would  place  on  pro¬ 
cess — especially  software  process.  This  led  to  the  definition  of  detailed 
process  research  themes.  Each  detailed  theme  was  specified  in  terms 
of  its  objective,  the  problems  it  was  addressing  and  specific  research 
questions.  The  detailed  themes  and  research  questions  across  all  the  sce¬ 
narios  were  assessed  to  identify  themes  and  questions  that  were  com¬ 
mon  to  many  different  scenarios.  Thus  we  aimed  to  identify  the  research 
themes  that  were  most  likely  to  be  of  value  irrespective  of  the  actual 
outcome  of  social,  political,  and  technological  trends. 


In  this  section,  we  present  synopses  of  the  plausible  scenarios,  with 
an  invitation  to  view  the  full  descriptions  at  a  later  time,  after  this 
framework  has  been  converted  to  an  online  forum.  Just  as  no  one  can 
accurately  predict  the  future,  a  scenario  developed  during  the  work  of 
this  group  might  not  apply  closely  enough  to  your  anticipated  needs. 
Readers  may  prefer  to  construct  their  own  scenarios  that  capture  the 
world  in  which  the  processes  your  organization  needs  must  operate  ef¬ 
ficiently  and  effectively. 


Envisioning  a  World  at  War 

(An  environment  of  continual  war  reshapes  the  meaning  of  survival.) 

The  causes  are  many:  conflicts  due  to  regional  tensions. .  .the  omnipresent 
threat  of  a  nuclear  attack  from  one  or  more  rouge  countries. . . rising 
terrorism. .  .a  failed  Iraq  war,  with  turmoil  erupting  throughout  the  Middle  East 

The  effects  are  staggering:  crude  oil  at  $200  per  barrel,  triggering  gas  prices  at 
the  pump  of  $1 0/gallon  in  the  United  States  and  3  to  5  times  that  elsewhere 
in  the  world. .  .every  major  commercial  airline  company  bankrupt. .  .economic 
investment  drastically  reduced. .  .the  U.S.  and  global  economies  on  the  brink  of 
collapse. 

In  this  desperate,  anxious  world,  the  highest  priority  assets  are  sources 
of  supply  for  food,  water,  and  energy,  national  and  international  critical 
infrastructures,  and  complex  systems  of  systems  including  the  Internet. 

How  will  people  adapt?  Technology  evolve?  Organizations  be  sustained? 
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Jurassic  Park 

(Reconfigurable  packs  of  capitalists  dominate  the  business  land¬ 
scape.) 

Swiftly  moving  Corporate  Velociraptors  thrive  in  the  creative  business  chaos 
spawned  by  rapidly  advancing  technology  that  has  outstripped  the  efforts 
of  regulatory  forces  and  confounded  big  business.  Globalization  is  emerging 
from  the  "bottom  up,"  and  traditional  value  chains  have  been  replaced  by 
dynamic,  constantly  collapsing  and  reforming  value-creation  networks. 

These  agile  businesses  move  for  a  fast  "kill"  in  highly  mobile  and  dynamic 
groups,  where  the  partners  of  today  may  be  the  predators  of  tomorrow. 
Mainly  virtual  enterprises,  these  organizations  define  their  own  mode  of 
working  and  configure  the  infrastructure  they  need  for  their  tasks.  This  makes 
them  better  adapted  to  a  world  in  which  less  and  less  reliability  is  offered  by 
global  infrastructures  which  once  provided  stability  and  growth. 

In  this  age  where  innovation  and  opportunism  dominate,  how  can  processes 
be  continually  refashioned  while  remaining  effective? 


The  Golden  Age  of  the  Consumer 

(End  users  force  changes  in  how  software  is  developed.) 

Consumers  have  spoken  and  systems  and  software  engineering  organizations 
have  heeded  their  concerns.  Consumers  get  precisely  what  they  want 
(demand)  in  a  global  domain  where  solutions  automatically  handle 
geographical,  cultural,  language,  and  psychological  differences  between 
individuals  in  the  global  consumer  and  end  user  community. 

Around  the  globe,  consumers  communicate  their  needs  through  the  World 
Technology  Organization  (WTO),  which  regulates  all  publicly  accessible 
technological  products  having  a  registered  user  base  exceeding  500  users 
per  country.  All  WTO  member  countries  (120  and  counting)  must  abide  by 
an  extended  set  of  global  laws,  one  of  which,  the  Systems  Equality  Law, 
states  that  "all  systems  must  provide  form  and  function  to  each  user  in  a 
manner  which  neither  discriminates  on  the  basis  of  culture,  language,  sex,  or 
psychological  difference." 

One  interesting  development  is  that  every  user  of  WTO-compliant  systems 
has  a  profile  implant  that  broadcasts  a  psychological  profile  together  with 
cultural,  gender  and  language  details.  All  WTO  systems  then  present  an 
interface  based  on  this  profile — gone  are  the  days  of  navigating  through  a 
Web-based  application. 

Will  organizations  follow  the  WTO  model? 


IPRC  Framework  |  Sponsor  Statements 


87 


8 


Cybercrime  Pushes  Economic  Activity  to  the  Brink 

(Prevalence  of  cyber  threats  causes  enterprises  to 
reconsider  connectedness.) 

The  great  engine  of  economic  activity,  the  Internet,  spawns  the  seeds  of  its 
own  undoing.  Cybercrime  has  become  a  major  problem. 

Cyberattacks  by  worms  and  viruses  cause  millions  in  damage;  hackers  cripple 
large  financial  companies;  and  "phishing"  (sending  e-mails  that  masquerade 
as  requests  for  details  like  account  numbers  and  passwords  from  a  genuine 
commercial  organization),  "pharming"  (attempts  to  obtain  user  personal 
information  by  use  of  malicious  code)  and  "poisoning"  (attacks  on  the  domain 
name  server  that  redirect  a  browser)  are  increasingly  common. 

It's  clear,  then,  that  Information  Technology  (IT)  can  be  used  in  many  ways  to 
further  criminal  activities.  In  addition,  more  complexity  in  IT  systems  means 
more  system  flaws,  which  are  often  exploited  to  support  security  attacks. 

Will  there  be  a  time  when  people  abandon  the  Internet  altogether? 


Small  Teams,  Big  Needs 

(Today's  agile  methods  become  tomorrow's  organizing  principles.) 

What  began  years  before  as  a  product  strategy,  planned  obsolescence, 
threatens  to  over  take  the  business  world.  We  have  become  a 
throw-away  society. 

Groups  continually  form  companies,  and  then  quickly  and  abruptly  dissolve 
them.  Employees  work  for  a  company  for  less  than  a  year,  on  average. 
Everyone  is  self-employed,  contracting  themselves  to  teams. 

Regulators  have  given  up  trying  to  control  the  businesses  in  this  world.  While 
this  frees  companies  and  allows  a  sort  of  "molten  lava"  approach  to  flourish, 
it  also  means  that  customers  must  accept  the  risk  associated  with  using 
cheap 

(or  free)  software-intensive  systems  that  may  or  may  not  always  work  as 
promised. 

What  form  of  software  processes  can  enable  rapidly  constructed  teams  to 
start-up  and  working  together  productively  as  quickly  as  possible? 
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PandemicTriggers  Global  Instability 

("Community''  is  possible  only  through  the  Internet.) 

As  a  global  pandemic  of  avian  flu  threatens  to  bring  on  a  second  dark  age 

•  One-third  of  the  exposed  population  recovers  quickly. 

•  One-third  recovers  after  a  long  period  of  illness. 

•  One-third  dies. 

Also,  borders  close,  all  public  assembly  is  banned,  hospitals  are  overwhelmed,  and 
vaccines  are  not  available. 

Survivors  brace  for  the  impending  collapse  of  civilization.  Food,  water,  and 
energy  supplies  are  of  primary  importance.  Communications  channels  such  as  the 
Internet  become  the  main  means  for  people  to  collaborate  on  improvised  survival 
techniques  like  self-made  isolation  suits,  water  collection  and  purification,  and 
wind-generated  electricity.  Commerce,  even  school  and  church  attendance,  are 
possible  only  via  the  Internet. 

Given  the  supreme  importance  of  the  Internet  to  this  society,  what  processes  keep 
it  robust  enough  to  play  this  role? 


Winning  the  Cyberwar 

(Software  engineering  overcomes  cyber  attacks.) 

Unknown  cyberthreats  from  unknown  sources  are  dynamic  and  constantly 
changing.  The  highest  priority  assets  are  national  and  international  critical 
infrastructures  (complex  systems  of  systems),  including  the  Internet. 

Industry,  academia,  and  government  finally  agree  that  there  is  a  common  threat 
and  that  requires  collaboration  to  reduce  this  threat.  Governments  make  sufficient 
funds  available  to  support  basic  research.  Industry,  academia,  and  government 
groups  (e.g.,  US  Department  of  Defense)  collaborate  to  perform  necessary 
fundamental  research  and  exploit  that  research  in  commercial  and  military 
products  and  services. 

The  software  engineering  workforce  is  populated  with  a  sufficient  supply 
of  software  engineers  and  technicians  who  hold  credentials  in  information 
infrastructure  assurance  and  survivability.  Courses  and  other  learning  materials 
leading  to  the  knowledge  and  skills  needed  for  the  credentials  are  widely  available 
from  colleges  and  universities,  technical  schools,  junior  and  community  colleges, 
and  commercial  training  organizations. 

What  changes  to  systems  management  practices  are  needed  to  design,  implement, 
and  operate  networked  systems  that  can  recognize,  resist,  and  recover  quickly  from 
attacks? 
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It's  a  Small  World  After  All 

(Small,  agile  companies  and  collaborative  ventures  dominate  the 
software  industry.) 

A  strong  interplay  between  multiple  business  domains  results  from  a 
trend  towards  ever  increasing  levels  of  business  diversification.  Large 
corporations  move  into  new  domain  hybrids,  looking  to  capitalize  on  the 
blending  of  their  traditional  business  expertise  within  other  domains.  Smaller 
businesses  cooperate  to  channel  their  combined  forces  and  expertise  into 
new,  innovative  means  to  provide  market  differentiation  and  competitive 
advantage. 

How  will  processes  adapt  to  support  the  specific  blends  of  skills  from 
different  problem  domains? 


Multidisciplinary  Convergence 

(Multidisciplinary  convergence  isn't  seen  in  products, 
but  in  the  processes  to  develop  those  products.) 


This  multidisciplinary  convergence  is  visible  in  professional  journals  and 
meetings. 

From  Synergy  Week  magazine,  January  4,  2015: 

Conference  Announcements  of  Note: 

^IThe  5th  International  Conference  on  Bioware  has  announced  its  call  for 
proposals.  Submitted  papers  should  focus  on  one  or  more  of  the  following 
themes: 

•  Cellular  computing/medical  device  interfaces 

•  DNA/neural  network  integration 

•  Microsystems  dynamics 

•  Zero-gravity  cellular  computing  environments 

•  Eco-ware:  integration  of  global  bio,  hardware,  and  software  applications 
to  effect  ecological  stability 

•  Borg-isms:  challenges  and  solutions  in  integrating  self-aware  medical 
devices  intohuman  systems 

How  will  processes  be  defined  to  accommodate  the  multiple  contexts  of  the 
varied  disciplines? 
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Embedded  Software  Rules 

(Rapid  co-design  of  product  and  process  overhauls  development.) 

It's  2020.  Most  people  don't  need  a  computer  anymore,  because  physical 
devices  they  use  daily  have  the  intelligence  needed  to  achieve  the  desired 
tasks.  Embedded  software  is  so  prevalent  that  Embedded  Systems  and 
Software  (ESS)  Inc.  is  the  world's  largest  employer:  It  has  a  work  force  of 
about  1  billion  people. 


ESS  has  acquired  all  software  and  system  engineering  skills  from  Infosys, 
Microsoft,  IBM,  and  SAP,  and  many  more  organizations.  Still,  the  demand 
for  embedded  system  competence  far  exceeds  the  supply  of  competent 
developers  and  managers.  Consequently,  ESS  partners  with  companies 
that  produce  end-consumer  products  (such  as  cars,  clothes,  and  phones). 
For  each  product,  there  is  a  peer-team  from  ESS  and  the  product  company 
called  an  Embedded  Engineer  and  Domain  Expert  (EEDE)  that  develops 
the  whole  product.  The  EEDE  also  designs  the  development  process  using 
process  architecture  patterns,  product  architecture  and  requirements,  team 
competence,  and  location. 


How  will  the  team  composition  affect  the  definition  of  the  development 
process? 


Rain  Man  Transforms  to  Answer  Man 

(Data  mining  yields  usable  results  while  respecting  privacy.) 

It  used  to  be  that  an  Internet  user  who  wished  to  find  information  had 
so  much  information  available,  with  so  little  organization,  that  it  is  like 
conversing  with  the  fictional  movie  character  called  the  "Rain  Man": 
enormous  amounts  of  seemingly  related  information,  presented  out  of  context 
and  with  little  relevance  to  the  real  problem  at  hand. 

Now,  instead  of  a  conversation  with  the  idiot  savant  of  "Rain  Main,"  a  search 
on  the  Internet  for  information  is  a  conversation  with  the  "answer  man."  If 
you  ask  a  reasonably  well-framed  question,  you  will  get  answers  to  what  you 
intended.  If  you  ask  a  poorly  framed  question,  the  agent  will  engage  you  in  a 
short  clarification  dialogue  that  will  frame  the  query  adequately. 

Your  searches  generate  mountains  of  data  that  organizations  collect  with  the 
expectation  that  it  not  only  will  be  of  value  for  each  task  or  transaction  they 
perform,  but  also  will  become  highly  valuable  in  the  future.  The  rules  of  the 
road  for  the  privacy  and  appropriate  use  of  each  user's  data  are  managed  at 
a  fine-grain  level.  The  "answer  man"  agent  spends  its  spare  time  checking 
up  to  see  if  the  data  it  has  chosen  to  release  to  trusted  sources  is  somehow 
leaking  to  other  sources.  Retribution  for  those  who  leak  data  is  exacted  by 
large  groups  of  "answer  man"  agents  abandoning  or  boycotting  data  sources 
and  brokers  who  are  not  trustworthy. 

What  would  it  take  to  bring  the  answer  man  agent  about? 
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Open  Source  Comes  of  Age 

(Open  source  is  a  trusted  and  typical  part  of  the  software  industry.) 

Industry  has  adopted  open  source  solutions  for  incorporation  into  products  in 
all  application  areas. 

Those  software  engineers  not  employed  by  a  large  software  services 
corporation  have  moved  either  to  an  independent  consultant  role  or  to  small- 
and  medium-size  enterprises  who  produce  add-ons  to  open  source  products. 
Those  employed  in  the  software  services  corporations  are  all  convinced  of 
the  efficacy  of  the  product  quality  models,  process  improvement  models,  and 
existing  technology  evolution  paths  that  they  use  within  their  organizations  to 
develop  their  software.  These  technologies  have  all  been  used  successfully  in 
recent  decades. 

However,  the  corporations  also  make  use  of  open  source  software 
components  in  their  products,  although  they  do  not  contribute  open 
source  software  to  the  market.  They  do  contribute  funds  and  people  to  the 
development  of  open  source  software  by  other  groups  and  organizations.  This 
open  source  software  is  then  incorporated  as  components  within  their  own 
products. 

The  consultant  engineering  workforce  devotes  spare  time  to  the  development 
of  open  source  software  solutions.  Other  open  source  projects  employ 
a  single-site  development  model  in  which  there  is  no  large  community 
of  testers  but  rather  a  single-site  small  group  of  people  using  informal 
management  techniques  with  strong  configuration  management,  extensive 
unit  and  system  testing  and  planned  time  boxing  of  deliverables.  In  this 
model,  small  teams  with  informal  processes  rely  on  close  contact  using 
personal  and  electronic  communications. 

How  can  the  quality  issues  of  open  source  be  resolved? 
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BAE  Systems 


(Provided  by  Michael  D’Ambrosa,  the  BAE  Systems 
representative  to  the  IPRC) 

BAE  Systems  has  a  proud  heritage  that  dates  back  to  manned  flight  and 
early  wireless  communications.  From  the  early  days  when  pioneer  A.V. 
Roe  built  his  first  airplane  that  flew  little  farther  than  the  Wright  Brothers 
at  Kitty  Hawk  to  the  wireless  telegraph  operators  who  saved  hundreds 
by  getting  the  distress  signal  out  from  the  Titanic  to  the  perilous  World 
War  II  missions  over  Europe  in  Lancaster  bombers  to  the  inaugural  flight 
of  the  world’s  first  vertical  takeoff  fighter  to  the  total  systems  integration 
solutions  today’s  civil  and  military  customers  need,  BAE  Systems’  heri¬ 
tage  of  evolution  and  mergers  has  created  a  global  leader  that  has  a  proud 
past  and  an  eye  toward  the  future. 

Today  BAE  Systems  is  an  international  company  located  primarily  in 
the  UK  and  US  engaged  in  the  development,  delivery,  and  support  of 
advanced  defense  and  aerospace  systems  in  the  air,  on  land,  at  sea,  and 
in  space.  The  company  designs,  manufactures,  and  supports  military  air¬ 
craft,  surface  ships,  submarines,  fighting  vehicles,  radar,  avionics,  com¬ 
munications,  electronics,  and  guided  weapon  systems.  It  is  a  pioneer  in 
technology  with  a  heritage  stretching  back  hundreds  of  years. 

It  is  at  the  forefront  of  innovation,  working  to  develop  the  next  genera¬ 
tion  of  intelligent  defense  systems. 

We  are  proud  to  be  a  corporate  sponsor  of  the  International  Process 
Research  Consortium  (IPRC).  It  reflects  our  continuing  interest  in  pro¬ 
cess  improvement,  as  reflected  on  our  emphases  on  internal  research 
and  development,  our  training  programs  that  include  partnering  with 
various  universities,  and  our  relationship  with  the  SEI  and  other  process- 
focused  organizations.  We  have  developed  a  Virtual  University  to  ensure 
a  coordinated  and  integrated  approach  to  education  and  skills-capability 
development  within  BAE  Systems  and  to  maximize  leverage  from  our 
partnerships  in  education  and  academia. 

We  have  an  Advanced  Technology  Centre  with  a  focus  on  systems  engi¬ 
neering  concerns.  It  delivers  frontline  research  and  technology  to  BAE 
Systems,  its  joint  venture  organizations,  and  customers.  Its  role  is  to 
identify  and  develop  technologies,  systems,  concepts,  and  processes  that 
will  maintain  BAE  Systems’  position  as  a  leading  edge  organization  and 
enable  future  growth.  Its  world-class  scientists,  technicians  and  research¬ 
ers  work  across  a  wide  range  of  disciplines  including  micro  and  nano 
technology,  deep  space  telemetry,  human-machine  interfaces,  materials 
properties,  mathematical  modeling,  and  electronic  sensors. 
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We  also  monitor  and  evaluate  the  business  potential  of  technologies 
under  development  within  academia  and  other  research  institutes 
worldwide. 

To  further  underscore  BAE  Systems  focus  on  process  research  and 
performance  improvement  is  the  fact  that  for  more  than  1 0  years  BAE 
Systems  has  held  its  own  internal  SPIRE  (Supporting  Performance 
Improvement  Realization  within  Engineering)  Conference.  Originally 
focused  on  software  engineering,  it  has  expanded  to  cover  all  of  engi¬ 
neering  and  regularly  attracts  200-300  attendees.  It  is  modeled  after 
most  such  conferences  with  guest  lecturers  (from  the  U.K.  Ministry  of 
Defence,  the  U.S.  Department  of  Defense,  the  SEI,  etc.),  tutorials,  and 
three  to  four  parallel  tracks.  We  believe  that  we  are  the  only  defense 
contractor  to  sponsor  an  internal  conference  of  this  scale. 

In  summary,  BAE  Systems’  sponsorship  of  the  IPRC  is  in  line  with 
our  forward-looking  approach.  We  expect  to  continue  to  be  involved 
in  follow-on  activities  and  eagerly  await  the  results  of  some  of  the 
targeted  research  areas,  especially  those  that  extend  the  capabilities  of 
software  and  systems  engineering  to  effectively  manage  the  ever-in- 
creasing  needs  and  complexity  of  our  defense  systems. 
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Robert  Bosch,  GmbH 


(Provided  by  Stefan  Ferber,  the  Robert  Bosch  representative 
to  the  IPRC) 

Bosch  is  a  leading  global  supplier  of  automotive  and  industrial  tech¬ 
nology,  of  consumer  goods,  and  building  technology.  The  investment 
into  the  International  Process  Research  Consortium  (IPRC)  highlights 
our  long-term  interest  in  process  engineering  research.  Bosch  funded 
and  participated  in  this  consortium  because  in  our  experience,  the 
right  processes  make  a  difference  in  our  business. 

The  International  Process  Research  Consortium  was  a  small  research 
project  with  the  task  to  envision  a  technology  roadmap  for  “soft¬ 
ware  processes”  in  the  far  future.  The  scenarios  and  research  themes 
derived  in  this  highly  skilled  and  international  group  of  research¬ 
ers  and  practitioners  alike  are  a  base  for  our  own  internal  research 
agenda — not  only  in  process  engineering  but  also  for  software-inten¬ 
sive  systems.  The  intense  personal  dialog  in  the  IPRC  about  so  called 
“possible  futures”  and  their  implications  for  software  and  systems 
engineering  allowed  us  to  evaluate  experts’  intuition  about  what  is 
coming  next  as  a  challenge  for  our  company. 

The  IPRC  was  also  an  experiment  in  multicultural  teamwork.  The 
group’s  diversity  of  opinions  helped  us  to  think  beyond  our  own 
limits,  and  differences  in  style  are  still  deliberately  visible  in  the  final 
report. 

Having  one  final  framework  that  nicely  integrates  all  the  different 
views  is  just  the  starting  point.  The  identified  research  themes  need  a 
continuous  update  and  realignment  with  reality.  Now  Bosch  expects 
research  institutes,  technology-oriented  companies,  and  national  as 
well  as  international  funding  agencies  to  put  priority  on  the  derived 
topics.  Bosch’s  research  project  portfolio  in  2008  already  builds  on 
the  IPRC  results,  and  we  hope  to  answer  some  of  the  overwhelming 
number  of  research  questions  derived  by  the  IPRC. 
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Lockheed  Martin  Corporation  IS&S 

(Provided  by  Lynn  Penn,  the  LMC  representative  to  IPRC) 
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Lockheed  Martin  Corporation  (LMC),  an  advanced  technology 
company,  was  formed  in  March  1995  with  the  merger  of  two  of  the 
world’s  premier  technology  companies,  Lockheed  Corporation  and 
Martin  Marietta  Corporation. 

Headquartered  in  Bethesda,  Maryland,  Lockheed  Martin  employs  about 
140,000  people  world- wide  and  is  principally  engaged  in  the  research, 
design,  development,  manufacture,  and  integration  of  advanced 
technology  systems,  products  and  services.  As  a  lead  systems  integrator 
and  information  technology  company,  nearly  80%  of  Lockheed  Martin’s 
business  is  with  the  U.S.  Department  of  Defense  and  other  U.S.  federal 
government  agencies.  In  fact,  Lockheed  Martin  is  the  largest  provider 
of  IT  services,  systems  integration,  and  training  to  the  U.S.  government. 
The  remaining  portion  of  Lockheed  Martin’s  business  consists  of 
international  government  and  some  commercial  sales  of  our  products, 
services,  and  platforms. 

Integrated  Systems  &  Solutions  (IS&S)  is  one  of  five  principal  business 
areas  within  Lockheed  Martin.  IS&S  leads  the  corporation’s  systems 
engineering  and  integration  activities  for  high-value  network-centric 
information  and  intelligence  systems  that  support  missions  of  the  U.S. 
Depart-ment  of  Defense  and  other  national  security  customers. 

IS&S  was  proud  to  represent  LMC  on  the  International  Process 
Research  Consortium  (IPRC).  LMC  is  a  leader  in  many  technologies. 
However,  LMC  also  realizes  that  technology  alone  does  not  make  a 
program  successful.  It  takes  a  mix  of  technologies,  people,  and  process. 
LMC’s  appreciation  for  the  importance  for  process  is  integrated  into  its 
business  rhythm.  LMC  was  interested  in  the  IPRC’s  proactive  approach 
to  identifying  the  appropriate  process  research  needed  to  be  prepared 
for  tomorrow’s  technologies.  The  process  role  is  important,  and  LMC  is 
proud  to  be  a  member  of  an  international  group  of  experts  who  did  not 
feel  overwhelmed  with  the  goals  set  forth  for  them.  They  embraced  the 
challenge  and  produced  a  process  research  framework  for  the  future. 

LMC  looks  forward  to  continuing  this  relationship  through  periodic 
reviews  of  the  IPRC  frame-work  driving  research  through  the  next 
few  years.  LMC  is  also  actively  taking  part  in  the  first  research  project 
defined  by  the  IPRC,  which  will  study  and  document  improving 
processes  for  small  settings. 
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Software  Engineering  Institute 

(Provided  by  Bill  Peterson,  SEI’s  Software  Engineering  Process 
program  director) 

The  SETs  mission  is  to  advance  software  engineering  and  related  disci¬ 
plines  to  ensure  the  development  and  operation  of  systems  with  predict¬ 
able  and  improved  cost,  schedule,  and  quality.  Our  core  purpose  is  to 
help  organizations  like  yours  to  improve  their  software  engineering  ca¬ 
pabilities  and  to  develop  or  acquire  the  right  software,  defect  free,  with¬ 
in  budget  and  on  time,  every  time.  The  Software  Engineering  Process 
Management  Program  is  one  of  the  key  assets  within  the  SEI.  Our 
research  works  in  process  maturity  and  capability,  applying  best  pro¬ 
fessional  practices,  and  in  defining  measurement  frameworks  are  very 
well  known  worldwide.  In  our  execution  of  our  mission  worldwide,  we 
are  often  approached  by  individuals  and  organizations  wanting  to  col¬ 
laborate  with  us  on  work  of  mutual  interest.  This  often  happens  on  indi¬ 
vidual  projects,  but  we  also  work  on  broader,  long-range  research  topics 
that  don’t  have  an  immediate  project  focus. 

The  SEI  has  come  a  long  way  since  our  inception  in  1984,  and  with 
continued  success,  we  plan  to  be  supporting  the  software  and  systems 
community  for  another  20  years  and  more.  By  then,  our  world  and  our 
technologies  should  be  vastly  different  from  what  we  see  and  use  today. 
Preparing  for  that  world  requires  a  dynamic  blend  of  vision,  imagina¬ 
tion,  and  action.  Partly  in  response  to  the  interests  of  researchers  and 
sponsor  organizations  around  the  world,  and  in  fulfilling  our  mission 
and  purpose,  the  SEI  took  an  action  in  2004  to  prepare  for  the  next 
generation  of  software,  systems,  and  the  enterprise  processes  by  estab¬ 
lishing  the  IPRC.  As  this  document  describes,  we  invited  researchers  to 
join  and  bring  their  special  perspectives  on  current  and  future  issues  and 
topics  to  the  table.  We  also  worked  with  select  sponsors  of  the  research 
both  to  be  sure  the  IPRC  would  respond  to  their  real-world  needs,  as 
well  as  for  financial  support  of  the  research.  These  two  factions  of 
the  project,  as  well  as  all  of  the  individuals  who  participated,  melded 
quickly  into  a  harmonious  team  pursuing  a  wide  range  of  research  ideas 
and  users’  future  needs. 
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The  SEI,  with  its  reputation  as  a  “trusted  broker”  among  many  inde¬ 
pendent  or  even  competing  organizations,  is  a  unique  place  where  such 
a  team  can  form  and  get  leverage  from  the  values  that  all  players  bring 
to  the  table.  I  am  proud  to  have  been  part  of  the  formation  of  the  IPRC, 
to  have  helped  initially  sponsor  its  early  explorations,  and  to  also  bring 
additional  SEI  researchers  to  bear  at  the  request  of  the  other  sponsors. 
Not  only  did  the  SEI’s  Process  Program  provide  research  representa¬ 
tion,  the  Dynamic  Systems  and  Network  Security  Programs  were  asked 
by  the  IPRC  members  to  join.  The  SEI  was  thus  well  represented  across 
its  research  agenda. 

I  am  also  excited  about  the  future  work  of  the  IPRC  beyond  the  contin¬ 
ued  development  of  this  framework.  Future  directed  research  projects 
are  being  identified  that  provide  a  lower  level  of  research  and  develop¬ 
ment  in  areas  of  interest  to  subsets  of  the  IPRC  team.  Both  sponsors’ 
interests  and  researchers’  areas  of  expertise  will  be  used  to  identify  the 
directed  research  projects  that  they  wish  to  pursue  in  developing  and 
discovering  new  and  existing  solutions  to  future  problems  and  opportu¬ 
nities. 
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Tata  Consultancy  Service,  USA 

(Provided  by  Nidhi  Srivastava,  the  TCS  representative  to  IPRC) 


Global  operations  with  bases  in  39  countries,  a  clientele  of  best  in  class 
organizations,  and  71000  associates;  these  characteristics  reflect  the 
scale  of  our  operations  and  the  mindshare  we  enjoy  in  IT  consulting 
and  business  services.  As  businesses  have  evolved  to  find  new  ways  to 
optimize  operations,  TCS  has  pioneered  in  making  global  outsourcing 
and  managed  services  a  ubiquitous  operating  model  for  businesses.  Our 
learning  by  working  closely  with  global  customers  and  synthesizing 
best  of  breed  practices  has  given  us  a  natural  evolution  in  business  con¬ 
sulting  creating  the  mindshare  we  enjoy  today. 

TCS  commenced  operations  in  1968,  when  outsourced  IT  services  were 
relatively  obscure.  The  evolution  since  then  has  been  significant.  We 
learned  in  cycles  to  adapt  to  and  create  new  service  models  as  we  made 
customers  realize  benefits  of  changing  paradigms.  We  gained  a  ground¬ 
ed  understanding  of  diverse  business  challenges  that  confronted  global 
companies.  This  enabled  us  to  help  customers  to  achieve  business  ob¬ 
jectives  and  redefine  those  from  new  perspectives.  Our  role  in  this  value 
chain  has  been  multifaceted.  While  we  attained  insights  and  expertise 
in  diverse  industries  we  innovated  in  multiple  sources  of  technology  to 
make  IT  translate  to  business  benefits. 

Our  “Global  Delivery  Model”  is  the  strategic  service  delivery  concept 
that  differentiates  us  by  bringing  together  our  unique  capabilities.  These 
he  in  the  diversity  of  our  geographical  presence,  role  in  technology  eco¬ 
system,  and  mechanism  for  tapping  heterogeneous  sources  of  knowl¬ 
edge.  We  thereby  made  global  knowledge  sourcing  seamless  and  fluid 
by  bringing  in  cultural  and  operational  proximity. 

TCS  is  a  disciple  of  multiple  management  and  quality  philosophies. 

As  practitioners,  these  are  ingrained  in  our  services  and  operations,  be¬ 
ing  tested  by  time  and  scale.  TCS  was  assessed  at  CMMI®  Level  5  and 
PCMM®  Level  5  in  2004.  Our  Integrated  Quality  Management  System 
(iQMS)  instills  in  our  operations  the  quality  principles  found  within 
these  frameworks.  The  iQMS  framework  distills  best  practices  by  uni¬ 
fying  models  such  as  CMMI,  PCMM,  ISO  TL  9000,  and  AS9100  Rev 
B.  The  encapsulation  of  these  best  practices  and  principles  into  an  adap¬ 
tive  framework  helps  us  to  make  quality  in  TCS  transcend  from  tactical 
and  compliance  objectives  to  business  drivers. 
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As  a  sponsor,  TCS  has  introduced  the  “voice  of  the  customer”  to  IPRC 
Workshops  and  deliberations.  The  span  and  width  of  our  experiences 
has  driven  us  to  embark  on  research  to  define  next-generation  software 
processes  based  on  emerging  imperatives  and  business  factors.  Our  role 
in  the  International  Process  Research  Consortium  (IPRC)  is  important 
in  involving  the  community  in  this  quest.  TCS  will  continue  to  play  an 
active  role  in  promoting  the  IPRC  framework  by  fostering  exchange 
and  application  of  ideas  between  the  industry  and  academia. 
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University  of  Pittsburgh  Medical  Center 

(Provided  by  Alan  Lawson,  the  UPMC  representative  to  IPRC) 


During  the  past  decade,  UPMC  has  reshaped  the  health  care  landscape 
in  western  Pennsylvania  to  become  the  premier  health  system  in  the 
region  and  one  of  the  most  renowned  academic  medical  centers  in  the 
United  States.  As  a  $6  billion  organization  and  the  region’s  largest  em¬ 
ployer,  it  has  transformed  the  economic  landscape  as  well. 

Today,  with  over  40,000  employees,  UPMC  is  composed  of  19  hospitals 
and  a  network  of  other  care  facilities  across  western  Pennsylvania  and 
throughout  the  world:  doctors’  offices,  cancer  centers,  outpatient  treat¬ 
ment  centers,  specialized  imaging  and  surgery  facilities,  in-home  care, 
rehabilitation  sites,  behavioral  health  care,  and  nursing  homes. 

Over  a  period  of  rapid  growth,  UPMC  has  created  a  genuinely  inte¬ 
grated  health  care  delivery  system  and  aggressively  recruited  superb 
physicians  and  researchers  to  develop  internationally  renowned  centers 
in  transplantation,  cancer,  neurosurgery,  psychiatry,  rehabilitation,  ge¬ 
riatrics,  and  women’s  health,  among  others.  UPMC  has  also  invested 
significantly  in  information  technology  to  link  and  integrate  electronic 
medical  records  across  multiple  hospitals  and  care  settings  and  has 
invested  research  monies  to  seed  new  fields  like  regenerative  medicine 
and  biosecurity.  In  partnership  with  Italy’s  region  of  Sicily,  UPMC  has 
exported  its  expertise  internationally,  developing  a  hospital  in  Palermo 
to  provide  transplantation  and  other  specialized  services. 

UPMC  also  has  leveraged  its  world-renowned  clinical  services  and 
patient  care  reputation  into  one  of  the  country’s  fastest  growing  health 
insurance  plans  offering  an  array  of  commercial,  Medicare,  and 
Medicaid  products.  A  pioneer  in  the  management  of  chronic  diseases 
like  congestive  heart  failure,  asthma,  and  diabetes,  UPMC  Health  Plan 
has  been  recognized  as  one  of  the  best  insurance  plans  in  the  nation  by 
the  National  Committee  for  Quality  Assurance,  which  in  2005  ranked  it 
among  the  top  five  in  the  mid-Atlantic  region  for  effectiveness  of  care. 

UPMC’s  standing  as  one  of  the  nation’s  outstanding  medical  centers  is 
a  source  of  civic  pride.  The  region  benefits  from  UPMC’s  many  charita¬ 
ble  contributions  and  its  broad  array  of  community-based  programs  that 
are  designed  to  eliminate  health  disparities  in  underserved  populations. 
UPMC  also  has  made  a  major  commitment  to  arts  organizations  to  en¬ 
sure  that  the  region  has  not  only  an  outstanding  scientific  community 
but  an  outstanding  cultural  one  as  well. 
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A  passion  for  innovation  lies  at  the  heart  of  UPMC’s  success.  Through 
such  innovation,  UPMC  has  already  launched  a  portfolio  of  new  busi¬ 
nesses  in  information  technology,  biosecurity,  and  bio-medicine — all 
nurtured  from  its  core  service  lines.  UPMC’s  unique  strategy  of  com¬ 
bining  clinical  and  research  excellence  with  business-like  discipline 
translates  into  excellent  patient  care  for  western  Pennsylvanians  and  the 
promise  of  new  jobs,  new  businesses,  and  a  new  biotechnology-based 
economy  for  the  region. 

UPMC  has  been  a  leader  in  the  application  of  process  improvement 
disciplines  for  health  care  information  technology  with  a  commitment 
to  CMMI,  CoBit,  and  ITIL.  UPMC  has  pursued  a  fusion  of  the  long 
standing  and  highly  effective  tradition  of  process  and  quality  improve¬ 
ment  driven  by  world-class  clinicians. 

UPMC’s  CMMI  experiences  have  been  focused  almost  exclusively  on 
closing  the  gap  between  the  state  of  the  art  in  process  improvement 
and  the  state  of  the  practice  in  the  health  care  industry  and  UPMC. 
Adherence  to  existing  CMMI  approaches  has  been  highly  valuable,  yet 
areas  remain  where  the  model  is  not  completely  aligned  with  the  real¬ 
ity  encountered  when  serving  UPMC’s  clinicians  and  their  patients. 
UPMC’s  participation  in  the  International  Process  Research  Consortium 
(IPRC)  is  a  welcome  opportunity  to  help  the  process  improvement 
research  community  to  target  the  key  issues  that  limit  progress.  In  the 
health  care  industry,  challenges  remain  with  the  issues  of  complex  sys¬ 
tems  of  systems  from  multiple  suppliers  based  on  multiple  technologies 
that  must  serve  diverse  customer  usage  models  in  a  widely  dispersed 
geographical  area.  Clinical  and  business  solutions  are  needed  to  interop¬ 
erate  in  a  semantically  consistent,  process-efficient  manner  that  permits 
UPMC’s  best  people  to  remain  focused  on  patient  care.  UPMC  has 
great  expectations  that  pursuit  of  excellence,  on  behalf  of  the  patients 
and  communities  served,  will  be  a  common  goal  shared  by  process 
researchers.  Advances  in  the  state  of  the  art  of  process  improvement 
are  needed  to  address  specific  health  care  industry  needs.  It  is  UPMC’s 
pleasure  and  privilege  to  assist  the  IPRC  in  challenging  the  process  re¬ 
search  community  to  make  these  process  improvements  a  reality. 
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Appendix 


An  Instantiation  of  Research  Nodes  and 
Questions  for  Security 


Description  of  the  Research  Nodes 

This  description  interprets  and  expands  the  research  nodes  and  ques¬ 
tions  contained  in  Section  4  for  the  security  product  quality.  All  ques¬ 
tions  in  Section  4  remain  applicable  (substituting  security  as  the  product 
quality  of  interest)  and  are  not  repeated  here. 

Research  Node  S.1:  Establishing  Security  in  the  Systems  or 
Software  Development  Life  Cycle19 

The  objective  of  this  research  node  is  to  determine  the  extent  to  which 
processes  can  be  used  to  accurately  reflect  and  cause  the  instantiation  of 
required  security  product  quality  attributes  for  each  software  develop¬ 
ment  life-cycle  (SDLC)  phase. 

Research  questions  associated  with  this  node  include20 

S-1  How  is  security  expressed  in  each  phase  of  the  SDLC?1  What  are 
appropriate  expressions,  from  a  security  perspective,  of  how  the 
system  is  to  be  used  (see  SI  .4  misuse/abuse  cases)? 


S-2  What  processes  best  ensure  the  instantiation  of  established 
security  principles?2 


S-3  What  are  effective  processes  and  methods  that  ensure  that  known 
causes  of  security  vulnerabilities  are  not  present  in  each  phase  of 
the  SDLC?3 


What  processes  and  methods  can  be  used  to  accelerate  adoption 
of  known  methods  for  developing  low-defect-rate4 (and  thus  more 
secure5)  software?  (state  of  art/state  of  practice  gap) 


What  are  the  compelling  cost/benefit  arguments  to  do  so? 


Is  it  possible  to  build  and  verify  secure  software  and  systems  using 
S-6  agile  methods  [Beznosov  2005]? 


What  processes  can  be  used  to  ensure  that  security  requirements 
are  met  for  systems  composed  from  existing  components?6  For 
extensible  systems?7 


19  A  series  of  research  papers  that  address  requirements  engineering,  survivable  systems 
analysis,  models  and  templates  including  life-cycle  models  for  survivable  systems  is 
available  at  http://www.cert.org/research/papers.html.  Jarzombek  and  Goertzel  desribe 
a  comprehensive  treatment  of  security  as  a  product  quality  during  all  SDLC  phases 
[Jarzombek  2006]. 

20  The  questions  associated  with  this  example  instantion  are  identified  with  a  letter  S.  Else¬ 
where  in  this  report,  research  questions  for  the  theme  on  the  relationships  between  pro¬ 
cesses  and  product  qualities  are  identified  with  a  Q,  those  for  the  process  engineering 
theme  with  an  E,  for  managing  process  processes  with  a  P,  and  for  process  deployment, 
a  D.  The  questions  associated  with  the  technology  factors  in  the  section  on  the  effects  of 
emerging  technology  trends  are  identified  with  the  letter  T. 
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Research  Node  S.2:  Establishing  the  Relationship  between 
Process  and  Security  as  a  Product  Quality 

The  objective  of  this  research  node  is  to  establish  whether  there  is  a 
direct  relationship  between  security  as  product  quality  and  the  processes 
used  to  develop  the  product. 

Research  questions  associated  with  this  node  include 

S-8  What  is  the  role  of  process  in  ensuring  that  software  and  systems 
are  engineered  such  that  they  continue  to  function  correctly  under 
malicious  attack,  failure,  and  accidents?8 


Research  Node  S.3:  Measuring  and  Monitoring  Security 
Performance 

The  objective  of  this  research  node  is  to  establish  processes  to  accu¬ 
rately  capture  meaningful  measures  that  aid  in  determining  if  a  system 
is  meeting  its  security  requirements  (and  how  well)  during  all  SDLC 
phases. 


Research  questions  associated  with  this  node  include 


S-9  What  are  the  definitions  of  meaningful,  informative  security 

measures?  What  processes  are  needed  to  reliably  collect  these? 


S-10  What  measures  indicate  that  a  system  has  met  its  security 

requirements  for  each  SDLC  phase?  What  are  the  processes  for 
collecting,  analyzing,  and  reporting  these  measures? 


S-ll  What  measures  and  evaluation  processes  can  be  used  to 
determine  the  effectiveness  of  different  secure  software 
development  processes? 
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Research  Node  S.4:  Verification  and  Validation  of  Security 

The  objective  of  this  research  node  is  to  enable  managers  to  select  ap¬ 
propriate  assessment,  evaluation,  verification,  and  validation  processes 
to  confirm  the  achievement  of  security  requirements.  Process  selection 
is  guided  by  the  nature  and  complexity  of  the  system  being  constructed 
and  operated.  Methods  include  the  use  of  scenario-based  misuse/abuse 
cases. 


Research  questions  associated  with  this  node  include 

S-12  How  is  an  adequate  or  acceptable  level  of  security  determined, 
tested,  verified,  certified? 


S-13  What  processes  are  most  effective  for  assessing,  evaluating, 
verifying,  and  certifying  the  security  of  software  and  systems 
(including  those  provided  by  third  parties)? 


S-14  What  processes  and  methods  are  most  likely  to  reveal  security 
issues,  flaws,  and  vulnerabilities  during  each  SDLC  phase?9  And 
with  third  party,  open  source,  and  COTS,  or  other  component 
software? 

In  the  case  where  such  processes  already  exist  and  have  empirical 
evidence  to  justify  their  use,  what  can  be  done  to  accelerate  their 
adoption?  (state  of  art/state  of  practice  gap) 


S-15  What  processes  and  methods  allow  for  building  misuse/abuse 
cases  that  predictably  provide  evidence  that  security  product 
qualities  are  present? 


Research  Node  S.5:  Sustaining  Adequate  Security 

The  objective  of  this  research  node  is  to  enable  managers  to  select  pro¬ 
cesses  that  result  in  establishing,  sustaining,  and  evolving  an  adequate 
level  of  security  throughout  the  full  product  life  cycle. 


Research  questions  associated  with  this  node  include 

S-16  How  do  we  define  and  sustain  adequate  security  in  the  face  of 
increasingly  sophisticated  attacks  (attack  evolution),  technology 
evolution,  enterprise  evolution,  supply  chain  evolution,  and  the  like 
(all  sources  of  change  that  require  a  system  to  evolve)?  T 
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Research  Node  S.6:  Usable  Security 

The  objective  of  this  research  node  is  to  enable  users  to  effectively  ap¬ 
ply  and  use  required  security  mechanisms,  to  the  extent  these  are  visible 
to  the  user. 

Research  questions  associated  with  this  node  include 

S-17  What  user  interface  processes  and  methods  result  in  users 
applying  protection  and  security  mechanisms  routinely, 
automatically,  and  correctly? 


S-18  What  processes  result  in  minimal  to  no  user  involvement  in 
security? 


Research  Node  S.7:  Using  the  Marketplace  to  Drive  Adequate 
Security  (may  be  out  of  scope) 

The  objective  of  this  research  node  is  to  establish  processes  resulting 
in  a  consumer/customer  marketplace  that  will  not  purchase  software 
known  to  be  insecure. 


Research  questions  associated  with  this  node  include 

S-19  What  processes,  market  forces,  and  other  mechanisms  can 
be  used  to  require  organizations  that  produce  software  with  a 
significant  annual  volume  of  reported  vulnerabilities  to  improve 
their  products?11 
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Research  Questions  for  All  Themes 

Theme  Q:  The  Relationships  Between  Processes  and 
Product  Qualities 


Q-1 

How  can  a  consumer  with  limited  expertise  be  facilitated  to 
express  their  needs  in  a  sufficiently  prescriptive  way? 

Q-2 

How  do  we  specify  product  quality  requirements  in  general  and 
such  that  they  reflect  business  requirements? 

Q-3 

What  levels  of  product  quality  are  required  for  specific  types  of 
products?  For  specific  markets?  How  are  levels  expressed? 

Q-4 

How  do  we  specify  product  quality  requirements  in  a  quantifiable, 
measurable  manner?  (How  will  we  know  it  when  we  see  it?) 

Q-5 

Does  the  level  of  need  for  each  ility'  vary  by  business  domain?  If 
so,  how? 

Q-6 

Is  there  a  direct  relationship  between  desired  product  qualities 
and  the  processes  used  to  develop  the  product?  If  so,  how  is 
this  relationship  expressed  and  how  does  it  differ  based  on  the 
maturity  of  the  process  used? 

Q-7 

How  do  we  determine  the  relationship  between  process  and 
product  qualities? 

Q-8 

What  is  the  evidence  that  better  processes  are  instrumental  to 
delivering  better  products? 

Q-9 

Do  the  issues  for  product  quality  and  process  relationships  change 
for  composable  components  and  for  products  assembled  from 
composable  components? 

Q-10 

How  do  we  model  and  predict  product  quality — process 
relationships  in  the  context  of  different  maturity  levels? 

Q-11 

How  do  we  model  and  identify  the  tradeoffs  between  business 
requirements,  process,  and  product  qualities  (such  as  degree 
of  quality,  cost  (affordability),  schedule  (timeliness),  team 
competence,  risk)? 

Q-1 2 

How  do  we  model  and  identify  tradeoffs  among  required  product 
qualities? 

Q-1 3 

How  do  we  make  well-informed  decisions  using  the  results  of 
trade-off  analyses? 

Q-14 

How  do  we  select  processes  to  meet  specific  product  quality 
requirements? 

Q-1 5 

What  process  steps  significantly  influence  the  achievement  of  a 
specified  level  of  product  quality? 
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Q-16 

When  composing  systems  of  individually  certified  components, 
how  do  you  ensure  that  the  non-functional  product  quality 
attributes  such  as  security  are  achieved  in  the  composed  system? 

Q-17 

Can  quality  attributes  associated  with  intermediate  work  products 
be  used  as  indicators  for  quality  attributes  in  the  final  work 
product? 

Q-18 

If  not  in  absolute  terms,  can  intermediate  work  products  be  used  to 
inform  the  risks  of  failing  to  achieve  final  product  qualities? 

Q-19 

How  do  we  determine  and  specify  product  quality  acceptance 
criteria? 

Q-20 

How  do  we  test,  verify,  validate,  assess,  and  audit  that  product 
quality  requirements  are  met  in  the  product? 

Q-21 

Using  product  quality  and  process  measures,  how  can  we 
determine  that  we  are  on  track  to  achieve  the  desired  level  of 
product  quality  during  each  life-cycle  phase? 

Q-22 

Do  product  quality  verification  and  validation  approaches  scale 
up  and/or  change  in  the  context  of  different  product  architectures 
(e.g.  complex  systems  and  systems  of  systems)? 

Q-23 

What  is  the  role  of  automation  in  product  quality  verification  and 
validation? 

Q-24 

How  do  we  sustain  product  qualities  in  the  face  of  product 
operations,  product  requirements  change  (both  functional  and  non¬ 
functional),  and  product  evolution? 

Q-25 

How  do  we  measure  (assess,  audit)  that  product  qualities  continue 
to  be  present  at  the  required  level  and  degree  throughout  the 
product  life  cycle? 

Q-26 

What  is  the  role  of  automation  in  product  quality  sustainment? 

Theme  E:  Process  Engineering 

E-1 

How  can  usable  best  practices  be  identified? 

E-2 

What  kinds  of  processes  are  needed  for  value-creating  networks; 
virtual  teams;  partnering;  outsourcing,  multi-site  development, 
end-user  development? 

E-3 

How  can  we  align  processes  with  business  goals? 

E-4 

How  to  perform  a  gap  analysis  between  today's  state  and  a 
desired  future  state? 
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E-5 

How  can  we  best  specify  a  process? 

E-6 

How  can  process  definitions  be  packaged  together  with  a 
quantitative/qualitative  model  describing  their  behavior? 

E-7 

What  are  appropriate  process  notations? 

E-8 

Can  a  process  be  analyzed  to  determine  if  it  is  implementable? 

E-9 

What  process  evidence  is  required? 

E-10 

What  evidence  is  required  with  respect  to  process  risks? 

E-11 

How  can  this  evidence  be  specified  and  applied  to  the  selection, 
tailoring  and  integration  of  processes? 

E-12 

How  to  combine  evidence  and  "context"? 

E-13 

How  does  the  context  of  a  process  (e.g.,  organization  size,  culture, 
process  distribution)  influence  process  selection  criteria? 

E-14 

How  can  acquired  process  components  be  evaluated  and  certified 
(so  they  can  be  trusted)? 

E-15 

How  can  knowledge  from  the  related  areas  of  organizational 
and  behavioral  studies  be  incorporated  into  the  definition  and 
specification  of  processes  that  can  be  effectively  implemented? 

E-16 

How  can  we  assure  that  a  process  will  meet  product/project 
requirements  and  standard  compliance? 

E-17 

What  does  it  mean  to  certify  a  process  component  and  how 
could  this  be  achieved?  (For  example,  what  criteria  could  such 
certification  be  made  against?) 

E-18 

How  can  we  improve  process  specifications  based  on  feedback 
from  deployment  and  use? 

E-19 

How  can  the  value  of  a  process  be  determined  and  monitored? 

E-20 

How  can  the  quality  and  cycle  time  performance  implications  of 
process  decisions  be  evaluated? 

E-21 

What  effects  do  different  domains  have  on  the  selection  criteria 
for  processes?  The  critical  issue  here  is  the  need  for  a  clearer 
schema  for  classifying  and  categorizing  "domains."  There  is 
confusion  between  business  domains,  application  domains,  and 
industry  domains,  as  well  as  with  factors  incorporating  cultural 
issues.  Identification  of  critical  domain  characteristics  is  crucial. 

Once  these  are  known,  the  process  and  systems  engineering 
issues  can  be  addressed. 
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E-22 

How  can  we  define  the  scope  of  process  lines? 

E-23 

How  can  we  define  the  value  of  process  lines? 

E-24 

How  can  we  organize  processes  and  evidence  into  one  or  more 
process  lines  (similar  to  the  concept  of  product  lines)?  This 
includes  domain-specific  issues. 

E-25 

What  is  the  appropriate  degree  of  commonality  of  processes/ 
procedures  (e.g.,  across  multiple  sites,  disciplines,  and  cultures)? 

E-26 

How  to  understand  the  difference  between  different  domains  with 
respect  to  processes  and  measurement? 

E-27 

What  effects  do  different  domains  have  on  the  selection  criteria 
for  processes? 

E-28 

How  can  process  line  engineering  be  aligned  with  product  line 
engineering? 

E-29 

How  should  a  process  architecture  be  constructed  for  a  process 
asset  base? 

E-30 

How  can  the  right  process  elements  be  identified  in  the  asset  base 
for  a  specific  project  as  a  function  of  product  requirements  and 
team  competence? 

E-31 

Can  a  formal  approach,  based  on  a  sound  theoretical  basis, 
be  developed  to  address  the  tailoring  of  processes  for  specific 
implementations? 

E-32 

What  are  the  organizational  and  environmental  factors  that  affect 
tailoring  choices  (e.g.,  tailoring  for  small  enterprises  or  for  agile 
development)? 

E-33 

How  can  we  tailor  processes  with  predictable  effects  on 
efficiency? 

E-34 

How  can  processes  be  designed  for  easy  tailoring? 

E-35 

How  to  specify  processes  including  variability? 

E-36 

To  what  extent  is  it  possible  to  integrate  different  processes, 
in  particular  when  they  are  based  on  different  paradigms,  for 
example,  agile  processes  versus  more  waterfall-like  approaches? 

E-37 

How  can  we  harmonize  the  mental  model  of  sequential 
development  with  the  reality  of  continuous  iteration? 

E-38 

How  can  processes  be  packaged  together  with  a  quantitative/ 
qualitative  model  describing  their  behavior? 
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E-39 

Are  there  mechanisms  for  understanding  and  improving 
interoperability  between  processes  (composability  analysis:  pre/ 
post  conditions,  inputs/outputs,  styles,  assumptions)? 

E-40 

How  can  domain-specific  development  processes  (product 
dependent)  and  software/systems  development  processes  be 
synchronized? 

E-41 

How  does  the  context  of  a  process  (e.g.,  organization  size,  culture, 
process  distribution)  influence  process  evolution? 

E-42 

How  can  we  evolve  process  lines  based  on  deployment  feedback? 

E-43 

How  can  processes  be  made  sufficiently  adaptive  to  provide 
effective  support  in  responding  to  domain-specific  change? 

E-44 

What  process  infrastructures  are  appropriate  to  support  the  new 
technologies  and  concurrent  engineering? 

E-45 

How  do  we  create  easy  to  use  "experience  bases"  that  allow 
knowledge  to  be  stored,  updated,  and  accessed  by  developers  at 
varying  levels  (to  facilitate  the  continuous  evolution  of  problem 
domains  and  technical  skills  required  as  businesses  move  into 
new  hybrid  domains,  for  example)? 

E-46 

How  do  the  team  competencies  affect  the  engineered 
development  processes? 

E-47 

How  do  we  educate  people  for  process  engineering  in  general  and 
the  use  and/or  development  of  process  lines  in  particular? 

E-48 

How  do  we  educate  people  in  the  need  for  and  the  use  of 
evidence? 

E-49 

Which  activities  related  to  evidence  creation,  process  line 
engineering,  and  usage  can  or  should  be  automated? 

E-50 

What  automated  decision  support  is  useful  and  how  can 
automation  and  human  (educated)  creativity  be  balanced? 

E-51 

How  do  we  perform  process  model  configuration  management? 

E-52 

How  could  "process  patterns"  be  best  used? 

E-53 

What  is  the  role  of  process  simulation  in  providing  trust,  scaling, 
and  supporting  process  prediction,  selection,  tailoring,  and 
integration? 

E-54 

What  visualizations  can  support  process  management  (including 
different  views  for  different  stakeholders  and  different  business 
domains)? 
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E-55 

What  level  of  statistical  analysis  is  feasible  and  reasonable  to 
apply  to  process  management? 

E-56 

What  is  appropriate  support  for  automated  metrics  collection? 

E-57 

How  can  an  inference  engine  for  effective  display  and  retrieval  of 
processes  be  constructed? 

E-58 

How  can  we  use  process  evidence  to  derive  theories  about 
process,  product,  and  resource  dependencies? 

E-59 

How  do  we  define  intellectual  property  for  a  process? 

E-60 

How  can  we  integrate  process  engineering  and  workflow 
management  tools? 

Theme  P:  Managing  Project  Processes 

p-i 

How  do  we  select  the  best  possible  project  process  or  set  of 
processes  to  use?  The  latter  is  particularly  relevant  when  having 
different  processes.  Selection  of  the  project  process  is  a  key  issue 
to  be  able  to  achieve  project  goals.  Moreover,  it  must  be  possible 
to  adapt  the  process  to  different  types  of  applications,  project  size, 
and  so  forth. 

P-2  How  do  we  scale  processes  to  our  needs?  How  does  an  SME  know 
when  it  is  time  to  have  more  formal  procedures  in  place?  SMEs 
cannot  be  expected  to  have  well-defined  processes  in  place.  It  is 
hence  important  to  be  able  to  have  as  little  overhead  as  possible, 
but  still  sufficient  to  enable  the  delivery  of  high-quality  software. 
As  a  small  company  grows,  it  is  important  to  realize  that  the 
processes  used  have  to  change.  The  questions  are  when  to  change 
them  and  how  to  change  them.  It  is  easy  for  a  company  to  grow 
out  of  its  process  support. 


P-3  What  are  the  needed  competencies  for  the  required  tasks  on  a 
specific  project?  How  is  the  relation  between  competence  profile 
and  development  process  handled?  It  is  a  challenge  to  have 
the  right  mix  of  competencies  in  a  given  project,  in  particular 
when  certain  type  of  competence  may  be  needed  in  too  many 
projects  at  the  same  time.  It  is  also  a  particular  problem  for  small 
companies  where  often  the  same  person  must  take  on  multiple 
roles.  This  issue  becomes  an  even  larger  challenge  when  looking 
into  successive  nodes  within  this  theme.  Furthermore,  how  do  you 
"process-bond"  teams  that  are  brought  together  from  outside  the 
company  for  specific  projects? 
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P-4  How  do  individual  competencies  sum  up  in  the  team?  The 

distribution  across  site  means  on  the  one  hand  that  competencies 
at  different  sites  become  available,  but  on  the  other  hand  it  may 
be  difficult  to  get  what  one  wants  from  a  project  perspective.  This 
may  be  because  the  site  manager  has  many  projects  and  would 
like  to  staff  them  according  to  an  optimization  for  the  site  and  not 
for  a  project. 


P-5  How  do  we  make  optimal  use  of  available  competencies?  Different 
companies  mean  different  competencies.  How  do  we  effectively 
combine  competencies  that  are  available  in  different  companies? 


P-6  How  do  we  capture  and  share  experiences  across  sites?  Project 
work  means  building  experience,  which  is  difficult  to  share  in 
general,  but  becomes  even  more  complicated  when  the  team  is 
distributed  across  sites.  A  certain  understanding  or  experience 
may  come  at  a  site  internal  meeting,  but  how  is  this  experience 
communicated  to  others  not  at  the  meeting,  and  in  particular  if 
they  are  at  another  site? 


P-7  How  do  we  manage  development  between  different  locations? 

This  includes  managing,  for  example,  responsibilities  and  risks 
between  locations  (including  risks  inherent  in  distributing  in  the 
first  place).  Development  across  locations  is  in  many  cases  a 
necessity  for  large  project  and  large  companies.  This  means  that 
the  actual  distribution  of  the  development  must  be  managed.  What 
management  procedures  are  needed  to  manage  a  distributed 
project?  How  is  time  reporting  done?  To  what  extent  do  things 
have  to  be  done  in  the  same  way  at  the  different  locations? 


P-8  How  do  we  divide  the  work  effectively  and  efficiently  between 
locations?  The  work  should  not  only  be  divided  between  the 
locations,  it  should  be  divided  in  an  effective  and  efficient  way. 
This  includes  taking  the  current  architecture  into  account 


P-9  How  do  we  distribute  quality  requirements?  The  distribution  of 
work  often  means  that  functionality  is  distributed,  but  how  do  we 
handle  non-functional  requirements?  The  customer  or  market  has 
expectations  on  certain  quality  aspects  for  the  whole  system,  but 
when  distributing  the  work  it  may  mean  that  quality  requirements 
are  forgotten  or  it  is  very  difficult  to  break  them  down  to  parts  of 
the  software.  A  typical  example  is  how  to  handle  performance. 
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P-10  How  do  we  handle  different  time  zones?  Times  zones  are  obstacles 

for  communication,  but  also  an  opportunity  for  having  development 

24  hours  a  day.  This  includes  challenges  related  to 

•  How  do  we  handle  time  zones  and  how  do  we  effectively 
communicate  the  results  from  one  site  to  another?  How  do  we 
make  use  of  differences  in  time  zones?  This  question  is  about 
the  opportunities.  How  can  different  time  zones  help  us  develop 
software  more  efficiently?  Is  it  possible  to  work  24  hours  a 

day  with  development  and  pass  assignments  around?  Can 
technologies  such  as  Instant  Messaging  and  Voice  over  IP  make 
the  world  one  global  workplace? 

•  What  is  the  productivity  drag,  if  any  due  to  time  zone 
differences?  Is  the  productivity  different  in  different  sites? 

Why?  How  can  we  leverage  form  the  best?  Do  people  in 
different  cultures  make  different  types  of  mistakes? 

•  How  do  we  work  toward  a  joint  base  (configuration 
management)?  Many  configuration  management  tools  exist,  but 
are  they  able  to  cope  with  potential  issues  such  that  developers 
in  different  sites  wanting  to  work  on  the  same  component  at 
the  same  time? 

•  How  do  we  manage  different  cultures  and  time  zones  between 
different  companies?  Different  companies  imply  different 
processes.  Distributed  development  around  the  globe  also 
implies  different  cultures.  How  do  we  handle  the  mix  of 
cultures  and  mix  of  processes? 


P-11  How  do  we  handle  cultural  differences?  Cultural  differences  are 
a  fact.  The  challenge  is  first  to  be  aware  of  them  and  then  to  use 
them  to  our  advantage.  Can  formal  training  deal  with  the  above 
challenge?  How  do  we  educate  people  to  work  in  global  software 
development?  What  additional  knowledge  is  needed  to  ensure 
that  developers  can  work  in  a  global  environment?  It  does  not  only 
require  knowledge  about  software  development,  but  also  a  good 
understanding  of  the  challenges  with  global  development  and  with 
other  cultures. 


P-12  How  do  we  ensure  the  same  process  interpretation  in  different 
cultures?  Can  Enterprise  Process  Frameworks  address  the  issues 
of  multiple  process  styles?  How  heavy  or  light  should  an  enterprise 
framework  be  to  meet  business  objectives?  Even  when  having 
the  same  process  across  sites,  there  is  no  guarantee  that  it  is 
interpreted  in  the  same  way.  The  likelihood  is  higher  if  being 
within  one  country  and  with  people  with  a  similar  educational 
background,  but  it  becomes  much  more  a  challenge  when  having 
people  form  different  cultures. 


P-13  How  do  we  make  use  of  experiences  in  one  culture  in  another 
place?  It  is  well-known  that  it  is  difficult  to  share  experiences 
effectively.  However,  it  becomes  even  more  complicated  to  share 
and  use  experiences  in  one  culture  with  another  culture.  How  can 
this  be  done?  It  is  probably  insufficient  to  publish  experiences;  they 
have  to  be  transferred,  but  what  is  the  best,  most  effective  way? 
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P-14  How  can  the  end-user  perspective  be  captured  in  global  software 
development?  The  end-user  comes  further  away  from  the 
development  when  the  development  goes  global.  Thus,  it  may 
become  even  more  difficult  to  include  the  end-user  perspective  into 
the  development. 


P-15  How  do  we  make  processes  that  are  compliant  with  accepted 
standards?  Different  companies  in  different  countries  may  have 
different  standards  or  they  may  even  be  required  to  follow 
different  standards  due  to  legislation.  How  can  global  software 
development  be  conducted  with  different  standards? 


P-16  Are  global  standards  an  imperative  for  this  model  to  succeed?  Is 
it  possible  to  succeed  with  global  development  without  having 
common  standards?  Can  global  process  interface  standards  be 
developed? 


P-17  How  do  we  handle  the  use  of  different  project  processes?  Different 
locations  may  have  different  project  processes  or  at  least  different 
flavors  of  the  same  process.  This  poses  some  challenges  that  are 
reflected  in  the  remaining  questions  in  this  group. 


P-18  Is  the  output  from  one  process  producing  the  input  needed  in 
another  process,  or  are  the  differences  between  processes  at 
different  locations  a  problem? 


P-19  How  do  we  handle  interfaces  between  processes?  The  interface 
between  different  processes  in  distributed  development  must  be 
well  specified  and  the  output  from  one  process  should  be  the  input 
to  another  process. 


P-20  How  are  the  abilities  to  deal  with  multiples  process  style 
managed?  For  example,  is  it  possible  to  combine  a  waterfall 
approach  in  one  location  with  a  much  more  agile  approach  in 
another  location?  What  requirements  have  to  be  put  on  the 
development  to  ensure  that  we  obtain  one  integrated  system  at 
the  end  of  the  day? 


P-21  How  can  we  handle  companies  with  different  process  capability? 
How  do  customers,  suppliers,  and  vendors  with  different  process 
capabilities  work  effectively?  Is  the  process  capability  determined 
by  the  lowest  common  denominator? 


P-22  How  do  we  integrate  open  source  code  in  company-specific 

products?  Open  source  is  often  used  as  "good  example."  How  can 
software  from  open  source  be  used  in  commercial  context?  How 
can  we  learn  from  how  open  source  development  is  conducted? 


P-23  How  do  we  manage  virtual  teams?  How  are  virtual  teams  formed? 
How  are  competencies  found  and  combined?  A  virtual  team  is  here 
defined  as  a  team  of  individuals  working  together  in  a  specific  role, 
for  a  specified  time  and  with  a  specific  goal,  however  without  any 
supporting  organizational  structures. 
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P-24  How  do  we  identify  suitable  partners/subcontractors?  How  are 
the  relationships  between  different  roles  managed?  The  problem 
here  is  about  identifying  the  most  suitable  partners  to  meet  our 
goals.  This  includes  identifying  suitable  collaborative  partners  and 
subcontractors.  What  is  the  role  of  SMEs  in  these  relationships? 


P-25  How  do  we  put  requirements  on  subcontractors?  What  is  a 

minimal  set  of  requirements  when  subcontracting  internationally? 
This  includes  both  functional  and  non-functional  requirements. 


P-26  How  do  we  accept  components  delivered  from  a  subcontractor? 
Delivered  software  must  be  accepted.  How  do  we  perform 
acceptance  testing?  This  may  be  particularly  difficult  when 
only  part  of  a  system  is  delivered.  How  do  we  certify  a  specific 
quality  level  for  a  component  or  subsystem  of  the  final  system? 
Particularly  where  the  overall  product  qualities  may  not  be  directly 
traceable  to  that  'component'? 


P-27  Do  locations  need  a  primary  and  secondary  role  based  on 

competencies  within  a  company  too?  Should  one  location  act  as 
the  primary  location  and  other  work  as  subcontractors?  The  actual 
roles  and  responsibilities  must  be  clearly  defined.  Should  sites  be 
structured  for  growth  as  "Centers  of  Excellence"? 


P-28  How  should  relationships  between  partners  be  formulated? 

Joint  development  of  a  system  means  having  some  explicitly 
agreed-upon  relationship.  The  relationships  include  other  sites, 
outsourcing,  partner  development,  and  networks  of  partners. 

The  networks  may  be  formed  for  a  specific  project  or  may  be 
more  long-term  and  go  beyond  the  scope  of  any  one  project.  It  is 
important  to  decide  what  the  core  assets  are  and  what  is  suitable 
to  let  others  handle. 


P-29  How  can  we  handle  interfaces  among  several  small  companies? 
To  compete  with  larger  companies  we  may  see  networks  of  small 
companies  working  together.  They  are  often  less  mature  and 
they  may  not  have  well-documented  processes.  How  do  small 
companies  together  become  mature  and  able  to  handle  larger 
projects? 


P-30  How  should  a  process  for  collaborative  development  be 

formulated?  The  development  at  different  companies  requires 
some  process  for  the  actual  collaboration.  How  should  it  be 
handled? 


P-31  How  do  we  handle  change?  Requirements  change  during 

development.  This  becomes  much  more  cumbersome  when  a 
change  may  affect  not  only  one  company,  but  several.  What 
routines  need  to  be  in  place  to  handle  change  across  companies  in 
distributed  development? 


P-32  How  do  we  create  value-based  networks  of  partners  to  work  as 
peers  in  projects?  How  do  we  establish  shared  goals?  Different 
companies  may  have  different  goals,  but  to  be  successful  shared 
goals  are  needed.  Is  it  possible  to  agree  on  common  goals? 
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Theme  D:  Process  Deployment 


D-1 

What  infrastructures  have  been  tried  and  what  are  their  relative 
merits? 

D-2 

What  is  the  best  advice,  then,  on  which  infrastructure  fits  which 
(contingent)  context? 

D-3 

Where  should  the  infrastructure  sit  inside  the  served  organization? 
What  is  the  best  governance  model? 

D-4 

How  much  responsibility  for  the  adoption  cycle  is  it  appropriate 
and  effective  for  it  to  take  as  its  charter? 

D-5 

What  is  the  profile  of  the  best-suited  people  to  staff  the 
infrastructure  roles? 

D-6 

What  is  the  quantitative  relationship  between  investment  in 
infrastructure  and  the  achievement  of  adoption  goals? 

D-7 

How  do  organizations  work  (in  the  ways  relevant  to  adoption)? 

D-8 

How  do  organizations  change  and  improve? 

D-9 

Assuming  there  is  more  than  one  answer  to  the  first  question  and 
to  the  second,  how  do  we  map  the  ways  in  which  organizations 
work  to  the  ways  in  which  they  change? 

D-10 

How  do  we  estimate  the  resources  and  duration  needed  to  make 
planned  change? 

D-11 

Is  there  a  way  to  assess  the  degree  to  which  an  organization  is 
ready  to  make  a  change  to  an  extent  we  specify,  both  in  depth  and 
rate  (how  much,  how  soon)? 

D-12 

What  is  the  best  way  to  characterize  the  environment  of  the 
organization  that  is  relevant  to  understanding  and  tailoring  the 
deployment? 

D-13 

Is  there  a  difference  in  deploying  process  vs.  product? 

D-14 

What  is  the  best  way  to  align  and  increase  the  motivation 
and  ability  of  personnel  to  adopt  new  processes?  In  fact,  what 
motivates  change  for  each  type  of  change  target?  For  example,  are 
business  cases  relevant? 

D-15 

What  is  the  best  way  to  plan  the  tailoring  of  the  new  processes  in 
light  of  the  answers  to  all  of  the  questions  above?  To  tailoring  the 
organization? 
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D-16  What  organization-based  confidence  or  acceptance  criteria  ensure 
that  when  defined  processes  are  adopted,  the  desired  results 
will  be  produced?  (These  criteria  are  used  to  decide  when  the 
engineered  processes  produced  by  the  Process  Engineering  layer 
are  worth  deploying  in  the  organization.) 


D-17 

How  do  we  determine  and  coherently  express  understanding 
of  the  cause  and  effect  relationship  among  processes/process 
changes,  deployment,  and  product  outcomes? 

D-18 

What  explains  the  difference  in  the  rate  and  success  of 
deployment  (i.e.,  small  vs.  large,  government  versus  commercial, 
business  sector  differences,  etc.)? 

D-19 

What  are  they  key  strategies  and  tactics  to  motivate  process  use 
in  settings  where  process  adoption  has  failed  in  the  past? 

D-20 

How  can  we  best  formulate  process  needs  related  to  business 
goals? 

D-21 

How  should  value-based  processes  be  deployed? 

D-22 

How  can  we  best  construct  sets  of  processes  that  when  deployed 
will  deliver  desired  business  outcomes? 

D-23  How  do  we  express  process  needs  so  as  to  ensure  that  an 

organization  is  able  to  adopt  processes  meeting  its  needs,  e.g.,  by 
taking  into  account  the  relationship  between  process  capability 
and  organizational  maturity?  (Depending  on  an  organization's 
process  maturity-and  many  other  factors— it  may  be  able  to 
adopt  some  processes  better  than  others.  This  needs  to  be  taken 
into  account  when  selecting  and  tailoring  processes,  i.e.,  these 
constraints  need  to  be  expressed  when  stating  an  organization's 
process  needs.) 


D-24 

How  do  we  express  process  needs  so  as  to  ensure  that  deployed 
processes  will  be  able  to  respond  effectively  to  changes  in  the 
organization  and/or  the  business  environment? 

D-25 

Do  current  models  for  process  capability  and  selection  reflect  the 
linkage  to  risk  appetite  and  risk  tolerance  in  the  organization  or 
project? 

D-26 

What  are  enablers  and  barriers  for  process  adoption? 

D-27 

How  does  the  level  of  team  competency  affect  the  deployment  of 
processes? 

D-28 

How  do  we  ensure  that  required  competencies  can  be  rapidly 
developed  and  applied  in  the  deployed  processes? 

D-29 

What  are  the  key  factors  influencing  the  successful  deployment  of 
engineered  processes? 
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D-30  How  do  the  processes  for  expressing  deployment  requirements 
affect  the  need  for  transition  infrastructure?  Are  some  processes 
cheaper  or  faster  than  others?  Do  some  require  less  infrastructure 
and,  say,  more  active  participation  by  those  who  are  affected  by 
the  changes? 


D-31  How  do  we  ensure  that  required  competencies  can  be  rapidly 
developed  and  applied  in  the  deployed  processes  (JIT  Training)? 


D-32  How  can  we  support  team  performance  versus  individual 
performance? 


D-33  what  is  needed  to  develop  effective  teams  and  team  architecture 
(e.g.,  leadership  and  team  motivation)? 


D-34  How  do  you  make  the  transition  mechanisms  of  processes 

designed  for  one  context  easily  and  efficiently  adaptable  for  a 
new  context?  (i.e.,  when  an  organization  is  acquired  into  another 
enterprise)? 


D-35  How  can  CEOs  and  other  management  personnel  be  educated  and 
trained  for  process  deployment,  and  how  can  we  best  measure  the 
effectiveness  of  training? 

In  fact,  does  it  matter  how  effective  executive  sponsorship  is? 


D-36  How  do  we  define  measures  of  breadth  of  process  adoption  in  a 
particular  context? 


D-37  How  do  we  define  measures  of  depth  of  process  adoption  in  a 
particular  context? 


D-38  What  are  the  appropriate  success  criteria  upon  which  to  judge 

deployed  processes? 

•  Can  we  develop  a  generally  accepted  set  of  measures 
describing  the  effectiveness  of  deployed  processes  and  the 
impact  of  improvement  actions? 

•  Process  reality  is  different  for  different  organizations;  how 
does  this  affect  our  understanding  of  success  criteria? 

•  What  is  the  difference  between  documented  and  practiced 
processes?  Is  the  development  performed  according  to  the 
documented  processes  or  how  does  the  actual  development 
differ  from  the  documented  process  within  an  organization? 


D-39  What  level  of  automation  can  be  introduced  to  facilitate  the 
assessment  process? 


D-40  How  can  we  best  validate  processes,  to  confirm  fitness  for 
purpose  and  return  on  investment? 


D-41  What  aspects  of  process  capability  are  important  and  relevant  for 
assessment? 


D-42  How  can  we  be  sure  that  the  results  of  process  evaluation  really 
reflect  the  ability  to  satisfy  business  needs? 
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D-43  What  lessons  can  be  learned  from  different  forms  of  process 
evaluation? 


D-44  How  can  we  improve  the  efficiency  and  effectiveness  of  process 
evaluation  approaches? 


D-45  How  do  we  convert  process  evaluation  to  a  continuous  activity 
that  can  be  effectively  automated? 


D-46  Since  all  measurement  is  taken  in  a  context,  what  is  the  best  way 
to  characterize  the  adoption  context?  What  is  the  best  way  to 
measure  that  context? 


D-47  How  can  adoption  effectiveness  be  compared  across  organizations 
or  between  any  two  organizations?  That  is,  how  do  we  know 
which  aspects  of  deployment  to  transfer  from  one  organization  to 
another?  How  do  we  know  what  we  are  supposed  to  learn  from 
each  other? 


D-48 

What  is  the  impact  of  changes  in  the  organization  and  the 
business  environment  on  the  process  needs  and  capabilities  of  the 
organization? 

D-49 

How  do  we  best  monitor  on  a  continuous  basis  the  capabilities 
and  everyday  use  of  deployed  processes? 

D-50 

How  can  we  best  monitor  the  ongoing  returns  on  investment 
on  process  deployment  and  improvement?  Is  ROI  an  important 
indicator? 

D-51 

What  is  the  most  effective  way  of  identifying  and  addressing 
changes  in  competencies  driven  by  changes  in  the  organization 
and  business  context? 

D-52 

How  can  we  identify  causes  of  variations  in  performance  and 
capability  so  as  to  initiate  effective  process  evaluation? 

D-53 

How  do  we  create  easy  to  use  "experience  bases"  that  allow 
knowledge  to  be  stored,  updated,  and  accessed  by  developers  at 
varying  levels? 

D-54 

How  can  concepts  like  network-centric  knowledge  management 
(could  be  social  and/or  infrastructure)  be  employed  to  leverage 
process  knowledge  across  multiple  contexts? 

D-55 

What  kind  of  ontology  would  be  useful  for  classifying  process- 
based  knowledge  and  the  context  of  its  acquisition  and  use? 

D-56 

How  can  we  improve  the  design  and  use  of  process  asset 
repositories  to  support  effective  learning  from  experience? 

D-57 

Can  the  full  validity  and  usefulness  of  statistical  process  control 
be  demonstrated  in  relation  to  software  and  systems  engineering 
and  management  processes?  If  yes,  how  should  it  be  used  to  aid 
adoption? 
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D-58 

What  are  the  limitations  of  applicability  of  statistical  process 
control  in  this  context? 

D-59 

Can  the  collection  and  analysis  of  process  metrics  be  improved  to 
reduce  effort  and  increase  acceptance? 

D-60 

How  can  we  best  identify  gaps  in  the  extent  to  which  an 
organization's  processes  address  its  goals,  and  what  is  the  optimal 
mechanism  to  address  these  gaps? 

Process  Effects  of  Emerging  Technology  Trends 

T-1 

How  do  you  anticipate  the  pressures  for  change  early  in  the 
system  life? 

T-2 

How  do  you  anticipate  requirements  volatility  and  unknown, 
unidentified  requirements  early  in  the  development  life  cycle? 

How  do  you  adapt  processes  to  allow  for  this? 

T-3 

What  metrics  can  we  develop  to  characterize  the  level  of  known 
requirements  satisfaction  and  the  level  of  dynamism  expected  in 
future? 

T-4 

How  to  you  characterize  the  level  of  known  requirements 
satisfaction  with  initial  build  and  the  level  of  dynamism  expected 
in  future? 

T-5 

How  do  you  support  a  requirements  management  system/process 
that  accounts  for  both  expected  and  unexpected  change? 

T-6 

How  do  you  instantiate  a  life  cycle  that  explicitly  recognizes 
incomplete  requirements  as  its  basis  for  initial  development? 

T-7 

How  do  you  instantiate  development  with  partial  requirements? 

T-8 

How  do  we  instantiate  processes  for  verification  and  validation  in 
initial  build  that  account  for  high  requirements  dynamism? 

T-9 

How  do  you  keep  verification  and  validation  tightly  coupled  to 
requirements  when  the  requirements  are  "a  moving  target"? 

T-10 

How  do  you  gather  and  filter  the  relevant  data  to  support  "the  next 
evolution"  in  your  requirements? 

T-11 

How  do  you  instrument  the  system  to  collect  information  that  will 
permit  appropriate  requirements  evolution? 

T-1 2 

How  do  you  model  the  eco-system  the  system  will  be  a  part  of? 

T-1 3 

How  do  you  measure  requirements  volatility  in  a  useful  way? 
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T-14 

How  do  you  sustain  and  evolve  an  eco-system  model  for  the  world 
the  system  lives  in? 

T-15 

What  is  important  and  essential  to  include  in  SoS  requirements 
analysis  models? 

T-16 

What  data  about  user  behavior  needs  to  be  collected  that  will 
help  infer  the  need  for  requirement  changes? 

T-17 

What  information  about  new  technologies  needs  to  be  collected 
to  help  infer  the  need  for  requirements  changes  and  the  impact  of 
the  new  technology  on  the  existing  system 

T-18 

What  replaces  "quality  =  conformance  to  requirements"  in  a 
system  where  continuous  requirements  evolution  is  present? 

T-19 

How  do  you  instantiate  an  assurance  strategy  (a  process  for  doing 
so)  given  that  a  subset  of  the  requirements  do  not  have  to  be  met 
all  the  time?  (i.e.,  in  a  SLA  context) 

T-20 

How  do  you  establish  a  release  cycle  that  balances  environment 
change  with  user  ability  to  integrate  or  deploy  the  new  increment? 

T-21 

How  do  you  anticipate  (determine)  the  pressures  for  change  at  a 
given  point  in  the  system  life? 

T-22 

What  are  the  relevant  utility  functions  for  establishing  release 
cycles? 

T-23 

How  do  you  recognize  when  to  re-architect  for  future  anticipated 
changes? 

T-24 

What  are  key  elements  of  the  design  or  architecture  that 
accommodate  operating  and  evolving  a  system  when  there  is,  and 
will  continue  to  be,  incomplete  system  state  knowledge? 

T-25 

For  a  given  system  type  and  operational  context,  can  we 
characterize  a  framework  of  where  and  how  incomplete 
knowledge  can  be  tolerated  or  dealt  with? 

T-26 

What  are  the  processes  for  determining  what  is  important  to  know 
or  not  know  about  system  state  and  configuration? 

T-27 

What  are  the  processes  for  using  knowledge  (complete  or 
incomplete)  about  system  state  and  configuration  for  system  build 
and  integration? 

T-28 

What  are  the  processes  for  determining  what  is  knowable  and 
unknowable  about  the  operational  state  or  configuration  of  a 
system  at  different  points  in  its  evolution? 

T-29 

How  do  we  determine  what  data  will  help  us  to  move  from 
"unknown  unknowns"  to  "known  unknowns"  to  "knowns"? 
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T-30 

How  do  we  determine  the  "shelf  life"  of  data  to  be  collected  at 
various  points  in  system  evolution? 

T-31 

What  are  approaches  for  understanding  which  elements  of  your 
configuration  can  be  ignored  at  any  particular  time? 

T-32 

How  do  we  provide  frameworks  for  eliciting  more  complete 
pictures  of  system  context  and  data  more  effectively? 

T-33 

How  do  we  effectively  collect  information  about  ambiguous  data 
and  data  types  to  help  reduce  that  ambiguity? 

T-34 

What  kinds  of  modeling  and  analysis  techniques  for  assurance 
help  us  to  characterize  the  level  of  completeness  of  our  assurance 
strategy  in  relation  to  the  knowledge  that  is  available? 

T-35 

How  do  we  provide  "data  cleansing"  to  deal  with  data  issues 
when  we're  acknowledging  incomplete  data  sets? 

T-36 

Given  that  conclusions  from  assurance  activities  are  necessarily 
probabilistic  in  nature,  what's  the  decision  process  to  use  them  for 
release  decisions,  etc.? 

T-37 

What  kinds  of  processes  can  be  used  for  relevant  quantitative 
assurance  when  system  state  or  context  of  use  is  unknown? 

T-38 

Are  there  requirements  approaches  that  will  reduce  integration 
and  assurance  challenges  for  heterogeneous  systems? 

T-39 

How  do  we  understand  the  indirect  and  cascading  effects  that 
are  inherited  from  component  systems  when  we  try  to  initially 
integrate  them? 

T-40 

How  can  we  create  market  and/or  regulatory  forces  that 
incentivize  vendors  to  create  products  that  are  verified,  secure, 
interoperable,  etc.? 

T-41 

How  do  we  change  the  education  system  to  prepare  future 
engineers  to  work  productively  in  this  environment? 

T-42 

How  do  you  decide  what  data  needs  to  be  collected  from  the 
heterogeneous  components  and  suppliers  for  acceptance  into 
integration? 

T-43 

How  do  you  build  the  trust  required  to  ensure  that  accurate  data  is 
being  passed  in  a  relevant  fashion  among  nodes  in  the  system? 

T-44 

How  do  you  collect  and  update  data  on  system  performance 
to  establish  service  quality  of  individual  components/new 
requirements? 

T-45 

What  kinds  of  processes,  models,  techniques,  etc.  help  you  to 
analyze  which  new  standards  actually  help,  rather  than  hinder, 
heterogeneous  system  evolution? 
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T-46 

How  do  you  analyze  data  collected  on  system  performance  to 
characterize  service  quality  of  individual  components  and  new 
requirements  needed? 

T-47 

How  do  we  analyze  components  to  determine  their  suitability  for 
inclusion  in  a  system  of  interest? 

T-48 

How  do  we  analyze  components  related  to  their  security, 
workflows,  user  roles,  decision  support  to  characterize  those 
aspects  across  the  system? 

T-49 

How  do  we  analyze  the  communication  paths  and  performance 
among  the  components  to  ensure  successful — and  eventually 
seamless — integration? 

T-50 

How  do  we  analyze  emergent  properties  (both  good  and  bad)  that 
are  present  in  the  heterogeneous  system? 

T-51 

How  do  you  establish  processes  that  can  navigate  different  legal 
environments  when  assuring  a  heterogeneous  system? 

T-52 

How  do  you  optimize  an  assurance  strategy  based  on  usage  of 
components  and  scale  of  integration  effort? 

T-53 

How  do  you  avoid  "big  bang"  integration  processes? 

T-54 

How  do  you  deal  with  asynchronous  upgrades  of  different 
elements  of  your  system? 

T-55 

In  an  environment  of  continuously  evolving  requirements,  how  do 
you  allocate  and  or  communicate  new  or  changing  requirements 
across  all  the  relevant  components? 

T-56 

How  do  you  manage  alignments  of  user  roles,  security,  workflows, 
decision  support,  etc  when  in  the  build/integration  process  when 
you  have  all  the  requirements  changing? 

An  Instantiation  of  Research  Nodes  and 
Questions  for  Security 


S-1 

How  is  security  expressed  in  each  phase  of  the  SDLC?  What  are 
appropriate  expressions,  from  a  security  perspective,  of  how  the 
system  is  to  be  used  (see  SI  .4  misuse/abuse  cases)? 

S-2 

What  processes  best  ensure  the  instantiation  of  established 
security  principles? 

S-3 

What  are  effective  processes  and  methods  that  ensure  that  known 
causes  of  security  vulnerabilities  are  not  present  in  each  phase  of 
the  SDLC? 
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S-4 

What  processes  and  methods  can  be  used  to  accelerate  adoption 
of  known  methods  for  developing  low-defect-rate  (and  thus  more 
secure)  software?  (state  of  art/state  of  practice  gap) 

S-5 

What  are  the  compelling  cost/benefit  arguments  to  do  so? 

S-6 

Is  it  possible  to  build  and  verify  secure  software  and  systems 
using  agile  methods? 

S-7 

What  processes  can  be  used  to  ensure  that  security  requirements 
are  met  for  systems  composed  from  existing  components?  For 
extensible  systems? 

S-8 

What  is  the  role  of  process  in  ensuring  that  software  and  systems 
are  engineered  such  that  they  continue  to  function  correctly  under 
malicious  attack,  failure,  and  accidents? 

S-9 

What  are  the  definitions  of  meaningful,  informative  security 
measures?  What  processes  are  needed  to  reliably  collect  these? 

S-10 

What  measures  indicate  that  a  system  has  met  its  security 
requirements  for  each  SDLC  phase?  What  are  the  processes  for 
collecting,  analyzing,  and  reporting  these  measures? 

S-11 

What  measures  and  evaluation  processes  can  be  used  to 
determine  the  effectiveness  of  different  secure  software 
development  processes? 

S-12 

Flow  is  an  adequate  or  acceptable  level  of  security  determined, 
tested,  verified,  certified? 

S-13 

What  processes  are  most  effective  for  assessing,  evaluating, 
verifying,  and  certifying  the  security  of  software  and  systems 
(including  those  provided  by  third  parties)? 

S-14 

What  processes  and  methods  are  most  likely  to  reveal  security 
issues,  flaws,  and  vulnerabilities  during  each  SDLC  phase?  And 
with  third  party,  open  source,  and  COTS,  or  other  component 
software? 

In  the  case  where  such  processes  already  exist  and  have  empirical 
evidence  to  justify  their  use,  what  can  be  done  to  accelerate  their 
adoption?  (state  of  art/state  of  practice  gap) 

S-15 

What  processes  and  methods  allow  for  building  misuse/abuse 
cases  that  predictably  provide  evidence  that  security  product 
qualities  are  present? 

S-16 

How  do  we  define  and  sustain  adequate  security  in  the  face  of 
increasingly  sophisticated  attacks  (attack  evolution),  technology 
evolution,  enterprise  evolution,  supply  chain  evolution,  and  the  like 
(all  sources  of  change  that  require  a  system  to  evolve)? 
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